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PREFACE 



This publication is the third in the series "UNEVOC Studies in Technical and 
Vocational Education" which aims at promoting international exchange of ideas, 
experience and studies relating to technical and vocational education. UNEVOC is 
the acronym of UNESCO's International Project on Technical and Vocational 
Education, which was launched in 1992. This Project focuses on the exchange of 
information networking and other methods of international co-operation between 
technical and vocational education specialists throughout the world. 

The activities of the UNEVOC Project include publication of a series of technical 
documents which will reflect experiences gained in field projects and by national 
institutions. This series is intended primarily for educational administrators and 
planners, teacher-educators, curriculum developers and all those interested in the 
current status and future development of technical and vocational education at the 
international level. 

The present document" A Guide for Evaluating Technical and Vocational Education 
Curricula" has been prepared under contract by the ILO International Training Centre 
in Turin and the author is Mr. Larry KENNEKE, Department for Vocational and 
Technical Education, State University of Oregan (USA). The purpose of this guide 
is to assist educators in the evaluation of vocational education curricula and will be 
useful to both experienced evaluators and novices. 

The views expressed in this guide are those of the individuals concerned and not 
necessarily those of UNESCO. The designations employed and the presentation of 
the material do not imply the expression of any opinion whatsoever on the part of 
the UNESCO Secretariat concerning the legal status of any country, territory, city 
or area or of its authorities, or concerning the delimitation of its frontiers or 
boundaries. 
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PREFACE 



This guide for eva.lu3.ting technica .1 a.nd voca.tiona .1 educa.tion 
curricula has been developed by the ILO Turin Centre and has been prepared 
under contract with the Division of Education Sciences and Technical and 
Vocational Education of Unesco, Paris. The manuscript was written by 
Larry Kenneke, Department Chairman, Vocational and Technical Education, 
Oregon State University, Corvallis, USA, visiting faculty member, ILO 
Turin Centre,, under direction of Pierre Martin, Promotion of Programmes 
Service, Turin Centre. 

The purpose of this guide is to help people evaluate technical and 
vocational education curricula. It has a wide potential for use. Because 
the guide contains procedures, checklists and forms, it can be used by 
persons conducting extensive evaluations of comprehensive curricula, or, 
it can assist evaluators as they collect, analyse and report information 
for a multitude of purposes: surveying attitudes, observing programmes or 
measuring achievement. This guide also explains concepts and vocabulary 
common to evaluation. It, therefore, can be used for both training and 
upgrading activities. This guide will be useful to experienced evaluators 
as well as those who are novices. 

This guide consists of six sections. The first describes the 
p^^ojs.e_, objectives, organisation and concepts upon which the guide is 
based. Procedures for developing an e valu ation plan are presented in 
section two. Information requirements for different kind of evaluations 
are explained in section three. Section four describes how to go about 
administering and supervising an evaluation. Four case studies 
illustrating evaluation of specific curricula appear in section five. The 
last section includes checklists, forms and vocabulary. 
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INTRODUCTION TO EVALUAT ION 



Pxirpose 



The purpose of this guide is to help you evaluate technical and 
vocational education (TVE) curricula. Procedures, checklists and forms 
are included to guide your evaluation activities. 



Objectives 

After reading this guide, you will know; 

. concepts and vocabulary common to evaluation 
. components of an evaluation plan 
. information needed to conduct an evaluation 

. administrative and supervisory procedures for managing an 
evaluation 

procedures for evaluating TVE curricula 
. methods of checking and recording evaluation activities 



Organisation and content 

The guide includes six sections. The first describes purpose, 
objectives, organisation and concepts upon which the guide is based. 
Procedures for developing an evaluation plan are presented in Section II. 
Information requirements for different kinds of evaluations are explained 
in Section III. Section IV describes how to administer and supervise 
evaluations. Four case studies illustrating evaluation of TVE curricula 
appear in Section V. The last section includes checklists, forms, 
vocabulary and supporting material. 



Reasons for evaluafiTig 

Evaluation provides decision-makers with information about TVE 
curricula. Administrators, specialists, teachers, students, employers and 
parents need information to make decisions. Credible data are needed by; 
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administrators when determining whether or not to continue, 
modify or terminate programmes; 

. specialists as they determine which educational practices work 
best ; 

. teachers as they measure student achievement; 

. students as they choose TVE programmes; 

. employers as they judge the appropriateness of curricula, and 

. parents as they help their children choose technical and 

vocational programmes. 

Only when people have access to credible evaluative data, can they 
make good decisions about TVE curricula and programmes. 



Evaluation models 



The contents of this guide were drawn from evaluation models shown 
in Exhibit 1.1. These models conceptualise the field, provide a working 
vocabulary and define evaluation tasks. A summary of similarities and 
differences among the models follows. Two models address a dicotomy of 
goals — one goal oriented (Bloom, 1981), and the other goal free 
(Scriven, 1967). The former assesses progress in view of the programme 
goals and the latter, programme effects based on criteria unrelated to 
stated goals. A third model ( Stuff lebeam, 1971) suggests that evaluation 
focus on four elements basic to decision-making. They are context, input, 
process and products. Context deals with deciding whether or not to offer 
a curriculum. Input addresses what resources and methods to use. Process 
evaluation focuses on effects the curriculum has had on students while in 
school. Product evaluation examines the curriculum's effect on former 
students. Morris and Fitz-Gibbons (1978) view evaluation as a process of 
providing information to people as they design curricula. People use this 
information to make decisions (Stufflebeeun, 1971). The kinds of decisions 
to be made depend on the stage of programme development. The four stages 
are assessing needs, planning, improving and judging. Needs assessments 
help set goal priorities. Programmes are planned using the priorities 
identified through needs assessments. Formative evaluations provide 
information for programme improvement while summative evaluations help 
people judge the worth of a programme. 
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Exhibit 1.1 



EVALUATION MODELS 
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Evaluation methods proposed in this guide reflect the models cited 
in Exhibit 1.1. They are based on the premise that decision-making is 
critical to the design, development and operation of curriculum 
( Stuff lebeam, 1971). Good decisions are based on comprehensive 
evaluations of curriculum. Comprehensive evaluations help you collect 
information and make decisions relating to: 

. Curriculum planning — whether or not to offer a TVE 
curriculum or programme; 

. Curriculum development — what goals, objectives and content 
should be included in a curriculum or programme: 

. Curriculum materials development — wh *:her or not 

instructional materials are of good quality; 

. Curricultim effects on students — whether or not content and 
skills are learned by students; and 

. Curriculum effects on former students — whether or not the 
curriculum affects the employability of former students 
(Finch, 1979). 

By using the process and procedures in this guide, you will be able 
to collect information on one, some or all of a curriculum's components. 

Exhibit 1.2 shows some of the components you could measure. 



Exhibit 1.2 

CURRICULUM C(»(PONENTS THAT CAN BE MEASURED 



. Need for the curriculum 

. Goals, objectives and content 

. Teaching-learning processes 

. Instructional materials 

. Supplies and equipment 

. Classrooms, workshops, and support facilities 

. Students 

. Teaching personnel 

. Budget 



The kinds of decisions that must be made wiljL determine which 
components you evaluate. 
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Key definitions 



Working definitions of evaluation terminology are required to 
clarify important issues . • Taken from the evaluation models summarised in 
Exhibit 1.1, the following definitions apply to this guide. 



DEFINITION OF EVALUATION 
Evaluation is a decision enabling process. 



Evaluations gather information for making decisions about 
curricultun, programmes, or instructional materials. Curriculum is defined 
to mean the following. 



DEFINITION OF CURRICULUM 

Curriculum is a training design or plan that defines: - 

aims, goals and objectives of an educational activity; 
ways, means and activities used to achieve aims, and 
methods and instruments needed to evaluate actions. 
(D’Hainaut, 1981) 



Because it has so many components, evaluating an entire curriculum 
is complex, time constuning and costly. 

For these reasons, evaluations often focus on components of a 
curriculum. Two components frequently evaluated are programmes and 
materials. A programme is defined as follows. 
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DEFINITION OF PROGRAMME 

A progranmie is oae component of a curriculum that 
addresses an instructional area or course 
agricultural mechanics, business procedures, child 
care, nursing or welding. 



A programme evaluation provides information on characteristics of a 
specific instructional area or course. For example, an evaluation of a 
commercial sewing programme may provide information on students, teachers, 
objectives, methods or achievements (Worthen, 1973)- 

Curriculum materials are also the focus of many evaluations. They 
are defined to mean the following. 



DEFINITION OF CURRICULUM MATERIALS 

Curriculum materials include all printed matter, 
audiovisuals and manipulative aids used for 
instructional purposes. 



They include instructional items like guides, books, circulars, 
worksheets, modules, multimedia packages and computer software. 

To provide people with appropriate information about curricula, 
programmes or materials, five questions must be answered: - 

. Who wants the evaluation? 

. What do they want to know? 

. What design will provide credible information? 

. What should be measured? 

How should information be reported? 



Who wants the evaluation? 



All evaluations are initiated at the direction or request of 
someone. A first step requires identification of those who initiated that 
request. While a directive may come from a single source, it is highly 
probable that a group of people prompted that request. For example, a TVE 
director may initiate a request for evaluation on behalf of an employer's 
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group, parent organisation or citizen's advisory committee. A project 
manager too, may initiate action to satisfy the information needs of 
funding agencies. 

These people are called the audience or target group. 
Administrators, managers, sponsors, employers, students, teachers and 
parents are but a few examples of people who request evaluations. Because 
each person's motives for requesting an evaluation are different, it is 
essential to determine who made the request. This knowledge will help you 
design and conduct an evaluation that provides^ the audience with 
information they want. 



What do people want to know? 

People want all kinds of curricular information. They may want to 

know: 

what a programme looks like, 

. whether objectives are being met, 

. how a teaching process is working, 

. whether programme graduates get jobs, or 
. if students are mastering job skills. 

To provide this information, you must know audience motivations — 
reasons for requesting the evaluation. 

People may simply want to learn about a programme. This requires 
information of a very general nature — description of aims, activities 
and outcomes. 

More often, requests are motivated by a need to make informed 
decisions. Administrators may need to decide whether or not to continue, 
modify or terminate a programme. Students may need to choose a programme 
of study. Employers may need to decide the fate of a cooperative work 

experience programme. These motivations require information quite 
different from the first example. Very specific information about student 
achievement, teaching methods and programme costs must be supplied. 

To determine the information needs of your audience, ask them these 
questions : 
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. What decisions do you want to make about the TVE curriculum? 

. What information will be needed to make these decisions? 

Finding answers to these questions will make your job easier as you 
plan and carry out the evaluation. 



Choosing an evaluation design 

Evaluations must have credibility. They must be believable to the 
evaluation audience. If evaluations are perceived as accurate, it is 
likely that findings will be discussed and used by the audience. What 
people accept as credible information is influenced by the evaluation 
design (Fitz Gibbon, 1978). 

A good design produces believable information because it provides 
comparative data. This allows results from the programme under evaluation 
to be judged against another programme or standard. It enables you to 
say, "compared to traditional instruction, modular instruction produced 
more capable students" or "compared to national safety and health 
regulation's, the mechanics workshop meets and in four cases exceeds 
minimum standards". 

These conclusions are more defensible than ones that merely report 
results without benefit of a comparison group or standard. Comparative 
data is obtained by using control groups or by making informal comparisons 
of results. 



Control group design. Exhibit 1.3 shows a true control group, pretest- 
posttest design. Students are randomly assigned to form two groups. One 
group receives a new method, material or programme. This is called the 
experimental (E) group. The other group, called control (C) does not 
receive treatment or gets an alternate one. Pretests are administered to 
see if both groups started out equal. If at the end of the treatment, the 
mean posttest score for the E-group is significantly higher than the mean 
posttest score for the C-group, the difference is attributed to the effect 
of the new method, material or programme. 




- 9 - 



Sbchibit 1.3 



Steps 



1 . 



. 



5 . 
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THE COWTROL GROtJP. PRETEST-POSTTEST DESICW 



Identify students, classrooms or 
workshops, some of whom will 
receive a new method or material. 



Pretest 


all students, 

or workshops. 


classrooms 






7 


Randomly 


assign 




some 


students. 



classrooms or workshops to the 
experimental and some to the control group. 



Administer new method or material 


to the experimental group. 


Control 


group receives an 


alternative 


or no programme. 





Posttest 


both 


groups 


with same 


test 


under 


7 

identical ' 


conditions. 
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Morris and Fitz Gibbon (1978) report three variations on this 
design. The true control group approach may be altered so both groups are 
measured only at the completion of the treatment. This variation is 
appropriate when a pretest might bias programme effects (when measuring 
Attitudes) or when a suitable pretest is not available. 



A non-equivalent control group, pretest-posttest is still another 
variation on the original design. Non-random groups are formed. This 
design is useful when students cannot be randomly assigned to programmes 
because they are bound by schedules, locations, prerequisites, or 
workshops. A third variation measures an experimental group and non- 
equivalent group at regular intervals. Both groups are measured before, 
during, and after a new programme, method, or material is initiated- 

Control group designs require assignment of students, classrooms or 
workshops to programmes; comparison of groups and rigorous control of 
variables. These controls enable you to say that evaluation results are 
due to the treatment and not other variables. True control group designs 
are highly credible means of determining whether or not a programme, 
method or material is effective. These designs indicate effectiveness. 
They do not tell you what deficiencies exist or how to correct them 
(Braskamp, 1984)- 



Informal comparison design. A second approach collects no formal 
comparison data. Instead, complementary information is gathered from 
different sources — student records, curriculum documents, health and 
safety standards, equipment and facilities specifications. This 
information is used cumulatively to draw conclusions about TVE curricula 
or programmes. Exhibit 1.4 shows four informal evaluation designs. For 
example, you may suspect that there is insufficient equipment in a welding 
workshop to meet programme objectives. To test your suspicions, you 
inventory existing welding equipment and compare the results with 
published workshop. ._.equipment__staniaxd^- The comparison may show a 
discrepancy between what equipment exists and what equipment should be 
available for a model programme, (Provus, 1971) • This discrepancy can be 
used to make recommendations for correcting equipment deficiencies. 
Corroborating evidence from different sources will strengthen informal 
interpretation of data. If you can get an expert to confirm the 
deficiency in workshop equipment, your conclusions will be all the more 
believable to an evaluation audience. 
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Exhibit 1.4 



INFORMAL DESIGNS USING CC^PLErnffARY INFORMAYION 



Programne Informal CoapleBcntary 

Information Comparison Information 




12 



Using complementary information to make informal comparisons of 
results helps determine curricular or programme deficiencies and how to 
correct them. This design is helpful when comparisons are not required 
for judging results — as in monitoring implementation of a new programme. 

An informal design is not satisfactory when assessing the quality of 
outcomes produced. This design does not provide answers to: "How good 
are the results?" and "Is the programme causing these results?". 

Whatever evaluation design you choose, remember that credibility is 
enhanced when comparative data is reported. Consider using a combination 
of designs. When appropriate, combine a control group design with an 
informal comparison of results. 



Deciding what to measure 



What you decide to measure depends on the kinds of decisions to be 
made by the evaluation audience. These decisions dictate the kinds of 
information you must collect. Knowing the information needs will enable 
you to ask appropriate questions and measure the right things. Exhibit 
1.5 shows five basic curricular concerns along with suggested items to 
measure (Lewy, 1977) • For example. "IF" the curricular concern is whether 
or not to offer a welding programme, "THEN" you might measure one or more 
of the following: 

. employer needs 

. student needs 

. supply and demand 

. goals and objectives 

. training costs 
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Exhibit 1*5 

MEASUREMENT DECISION MATRII 



IF THE CONCERN IS... 


THEN MEASURE. . . 


1. Should a curriculum be offered? 


. Employer needs 
. Student needs 
. Supply and demand for these 
workers 

. Goals and objectives 
. Costs of training 


2. What content should be offered? 


. Knowledge requirements of the 
area 

. Skill requirements of the job 
. Required attitudes 
. Related requirements 


3. Are curriculum materials of 

good quality? 


. Usefulness of materials 
. Acceptability of materials 
. Costs of materials 


4. ' What are curricular effects on 
students in-school? 


. Student achievement 
. Student performance 
. Student satisfaction 
. Quality of facilities 
. Adequacy of budget 
. Teaching quality 


5. What are curricular effects on 

former students? 


. Employability of graduates 
. Employer views of graduates 
. Graduates' satisfaction with 
TVE curriculum 

. Graduates' satisfaction with 
their jobs 



er|c 
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Exhibit l.S can also help you decide what to measure when you are 
concerned with content, materials, and curricular effects on students. 
When you want to know what content should be offered, consider measuring 
knowledge, skill, and attitudinal requirements (Item 2). Try measuring 
usefulness, acceptability and costs of materials when you are concerned 
with the quality of curriculum materials (Item 3)* Measuring student 
achievement, performance and satisfaction are three ways of determining 
curricular effects on in-school students (Item 4). Curricular effects on 
former students (Item 5) might be determined by measuring employers' views 
of graduates, graduates' satisfaction with the TVE curriculum or 
graduates' satisfaction with their jobs. 



Reporting the evaluation 

The real measure of an evaluation's success is the extent to which 
people read, understand and use findings given to them. A good evaluation 
report will increase the likelihood of achieving these outcomes. Report 
information that your audience wants to see. Present data in simple, 
easy-to-understand language. Use visuals, wherever possible, to 
communicate findings. Simple pie charts, bar graphs and tables are good 
ways to get your message across (Morris, 1978). 

Arrange information in a logical sequence starting with a background 
statement. Describe the people who requested the evaluation, what 
decisions will be based on the evaluation, what information was requested 
and the evaluation objectives (Exhibit 1.6). 



Exhibit 1.6 

EXAMPLE OF BACKOIOUND STATEMENT 

To determine whether or not to propose changes in curriculum, 
the vocational training board chairman requested information on 
programme objectives . He wanted to know if each programme 
objective addressed industrial training needs. The objective 
of this evaluation was to determine, for each vocational 
programme, which objectivesaddressed training needs. 
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Next provide a brief description of the data collection and analysis 
procedures. Describe the instruments and methods used to collect data. 
Follow with an explanation of techniques used to analyse the data (Exhibit 
1.7). 



Exhibit 1.7 

EXAMPLE OF DATA COLLECTION AND ANALYSIS 

A discrepancy analysis was used. Objectives of each 
vocational progranune were compared with counterpart industry 
training needs . Results of the comparisons were analysed for 
similarities and differences. Discrepancies between programme 
objectives and industry needs were recorded and reported. 



Present findings with the aid of visuals. one well-designed chart, 
graph or table will communicate more effectively than many words. Briefly 
explain the contents of each visual (Exhibit 1.8). 
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£xiiibit 1.8 

EXAMPLE REPORT OF FINDINGS 

Number of Discrepancies Between Industrial Training Needs 
and Objectives by Programme Areas 



N5 of N5 of 



objectives training 



Vocational Programme 


meeting 

needs 


needs Discrepancy 




Agriculture mechanics 


13 


15 


2 




Business practices 


10 


20 


10 




Construction 


15 


20 


5 




Food preparation 


12 


19 


7 




Health care 


15 


21 


6 




Totals 


65 


95 


30 




Thirty instances were found where vocational curriculum objectives 


did 


not address industry 


training needs 


. The greatest 


number 


of 


discrepancies were found 


in the business 


practices programme 


(10). 


The 


next highest number (7) 


was found in 


the food preparation 


programme. 



Health care, construction and agriculture mechanics each accounted for 
6, 5 and 2 discrepancies respectively. 




26 



17 - 



Conclusions follow findings. Concluding statements summarise 
findings and focus on important issues and points (Exhibit 1.9). 



Exhibit 1.9 

EXAMPLE OF CONCLUSIONS 

Vocational curriculum objectives are not meeting all of 
industry's training needs. The greatest discrepancy between 
industry needs and objectives appears in the business practices 
programme. Discrepancies are also evident in the school's four 
other programmes. 



Recommendations logically follow conclusions (Exhibit 1.10). They 
are suggestions for further study or action. 



Exhibit 1.10 










EXAMPLE 


- RECOMMENDATIONS 




1. 


Initiate continuing studies of 
industry training needs. 


the changing 


nature of 


2. 


Revise curriculum 
practices . 


objectives 


starting with 


business 



A complete example of a final report appears to Section VI. Use it 
and these examples to guide your report writing efforts. A final report 
is your method of communicating evaluation results to the audience. 
Prepare it well and your audience is likely to read it, understand content 
and apply recommendations. 

Plan your report before starting the evaluation. Knowing reporting 
requrements will guide your efforts throughout the evaluation process. 
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Evaluations provide people with information. People use this 

information to make decisions about TVE curricula. The evaluation 
strategy proposed in this guide was drawn from four theoretical models. 
These models promote understandings of the evaluation process and of the 
actions that must occur. Five actions underlying evaluations are: ~ 

. determine who requested the evaluation, 

. find out what they want to know, 

. choose an evaluation design, 

. identify what must be measured, and 
determine how to report information. 

Completion of these tasks leads to development of an evaluation 

plan. 
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SECTION II 

PLANNING AN EVALUATION 



o 

ERIC 



29 



20 > 



PLANNING AN EVALUATION 



Good evaluations do not just happen. They are carefully planned and 
executed. Quality evaluations are related to the amount and type of 
planning that has gone into them (Finch, 1982). 

Planning requires much time and coordinated effort. This requires 
definition of purpose and preparation of an evaluation plan. 



DEFINITION OF EVALUATION PLAN 

An evaluation plan is a framework for collecting, 
analysing and reporting data. 



This framework increases the likelihood that an evaluation will be 
successful. A good plan will result in an evaluation that provides 
decision makers with desired, credible and timely information. 

Advisory committees, citizen groups, teachers, and administrators 
can provide valuable planning assistance. Use them as you prepare the 
evaluation plan. Solicit their advice as you define the scope of 
evaluation; its measures, designs, procedures, and budget. Check with 
them as you complete each part of the plan. Those who have experienced 
similar evaluations or situations may suggest ways to improve your plan. 
Still others may serve as sounding boards against which you test your 
ideas about evaluation measures, designs and procedures. By involving 
others, you create interest and ownership in the evaluation. When people 
feel the evaluation is their creation, they will work to see it succeed. 



Planning process 



The evaluation planning process is illustrated in Exhibit 2.1. It 
involves completing five planning worksheets and one summary document. 
The worksheets are used to collect information on five aspects of the 
evaluation. 
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Worksheet number 



Worksheet title 



1 

2 

3 

4 

5 



Scope of the evaluation 
Progranune description 
Measures and designs 
Data concerns 
Budget 



Each worksheet contains a series of steps that ask questions about 
the programme or evaluation process — step 1.1, who wants to know about 
the programme?; step 1.5, what is/are the objective(s) of the evaluation? 

Information collected at each step of each worksheet is summarised 
and transferred to a one page summary document. This document and five 
supporting worksheets comprise the evaluation plan. The entire process is 
validated by making continuous progress checks with advisory bodies, 
citizens groups, teachers and administrators. 
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Exhibit 2.1 



EVALUATION PLANNING PROCESS 




o 
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Example evaluation plan 

An example plan illustrating the use of planning worksheets appears 
in Exhibit 2.2. Because evaluating an entire curriculum is so complex, 
the example plan addresses only one curriculum component. However, the 
process and procedures illustrated are applicable to the planning of 
comprehensive curriculum evaluations. 

The example describes a plan for evaluating the effects of modular 
instruction on student achievement in an electronics assemblers programme. 
A small ’’experiment" was selected to provide desired information. 

Information for the plan was collected using the five planning 
worksheets described in Exhibit 2.1. The data collected at each step was 
used to prepare the evaluation plan (as in Exhibit 2.2). An explanation 
of how to prepare a plan and worksheets follows. Worksheet preparation 
will be explained first. 



Scope of the evaluation 



Planning sheet 1 helps you describe five aspects of the evaluation. 
Begin by writing the name of the programme to be evaluated under step 1.1. 
Then identify the people who want to know about the programme (step 1.2). 
After you have identified the audience, find out what decisions they will 
make using evaluation information. You may want to ask them: 

What decisions do you want to make regarding the TVE 
curriculum? 

. ' What information do you think you will need to make these 
decisions? 

Enter decisions under step 1.3* Analyse the kinds of information 
your audience said they needed to make decisions. Make modifications in 
needs to ensure proper data collection. Describe the information needs in 
step 1.4« 
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£]diibit 2*2 

SAMPLE EVALUATION PLAN 

Person(s) responsible for the evaluation: Wiley, Suzuki, Kheir. 



1 . Evaluation scope 

The electronic assemblers programme adopted mastery learning. This 
required purchase of modules. Because modules are costly, officials need 
to know their effectiveness. The evaluation objective is to determine how 
effective modules are in helping students develop job competencies. 
Findings will help officials determine whether or not to continue, modify 
or terminate the mastery learning process. 

2. Programme description 

This ten week programme prepares students for work as electronic 
assemblers. It develops competencies necessary to read schematics, solder 
connections, strip and wrap wires, harness cables, and install mechanical 
components. Two experienced tradesmen teach the course using learning 
modules. These modules require slide projectors, instruction manuals and 
25 assembly stations. 

3. Measures and designs 

A pretest-posttest true control group design will be used to assess 
the impact of learning modules on student achievement. Students will be 
given a manufacturer's pretest on soldering. Two equivalent student 
groups will be formed by random assignment. Students in the experimental 
(E) group will spend two weeks using modules to learn soldering 
techniques. The control group (C) will learn to solder using traditional 
methods. At the end of two weeks, students will be tested to see if the 
module using (E) group has learned more than the control (C) group. 

4* Data concerns 



A pretest will be given on 1 April. Two weeks of instruction will 
be given using both modular (E-group) and traditional instruction (C- 
group). Both groups will be posttested on April 15» A t-test will be 
used to show significance of difference between mean scores of the E and C 
groups. A written report documenting findings, conclusions and 
recommendations will be completed no later than 11 May. 



5. Budget 
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Funds required, to carry out the evaluation as proposed equal 
US $ 1,720. Personnel will require US $ 1,550, with travel, test items, 
and communications requiring US $ 25, $ 85 , and $ 60 respectively. 
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PLANNING SHEET 1 



SCOPE OF THE EVALUATION 



STEPS 

1.1 

1.2 

1.3 

1.4 

1.5 



What programme is to be evaluated? 

Electronic assemblers 

Who wants to know about the programme? 

TVE advisory committee and administrators 

What decisions will be based on the evaluation? 

Whether or not to continue, modify or terminate the modular 
learning approach. 

What information must be given to the audience? 

Data on how effective modules are in helping students meet 
objectives . 

What is/are the objective(s) of the evaluation? 

To determine how effective learning modules are in helping 
students meet objectives. 



BEST COPY AVAILABLE 
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Answering the questions in steps 1.1 through I .4 enables you to 
state the evaluation objective(s) . The nature of the situation will 
determine how many objectives must be stated. Use as few objectives as is 
possible. One well-stated objective is better than many poorly defined 
ones. Objectives give direction to evaluation efforts. They are 
important in creating an environment conducive to evaluation. Evaluation 
objectives help in: 

. creating support for the evaluation, 

. selecting measures and designs, 

. assuring proper data collection and analysis, 

. informing participants of expectations, and 

. guiding evaluators and support personnel during the evaluation 
process . 

An evaluation objective is a statement of what will be done to 
secure information required for making decisions about a curriculum or one 
of its many components. Some example evaluation objectives follows: 

. to determine how effective learning modules are in helping 
students achieve objectives, 

. to determine, for each vocational programme, which objectives 
address industrial training needs, 

. to determine the need for a farm mechanic programme, 

to describe existing workshop conditions in relation to 
documented standards, 

. to determine programme effects on the employability of former 
students . 

Entering the evaluation objective(s) under step 1.5 completes 
planning sheet 1 . 

Prograaiae description 

Planning sheet 2 describes the curriculum, programme, method or 
materials being evaluated. When two programmes are compared, 

characteristics of both must be described. This enables you to measure 
exactly how one programme measures-up against an alternative. Complete a 
separate planning sheet for each programme. 
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PLANNING SHEET 2 




PROGRAMME DESCRIPTION 


Steps 




2.1 


What are the progranune outcomes? 




Students successfully completing the programme will be 
employable as electronic assemblers. They will meet industry 
standards for; - 




. reading schematics; 

. using hand and power tools; 

. soldering electrical connections; 

. harnessing cables; and 

. stripping wires. 


2.2 


What are the programme activities? 




Two teachers , one with 20 and the other with 8 years 
industrial experience teach the class. They use lectures, 
demonstrations and workshop exercises to impart knowledge and 
skills. Students observe techniques and than apply them to 
individual practice exercises. 


2.3 


What are the required resources? 




2 

A 200 m workshop is fully equipped with 24 work stations, 
lecture area and storage room . Chalkboards , flipcharts , 
overhead projector and job sheets supplement instruction. An 
operating budget of US $ 280 is available for each student. 
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Programme characteristics may be grouped into three categories: 
desired outcomes, activities, and resources. Desired outcomes may include 
aims, goals and objectives. List these under step 2.1 Programme 
activities are all actions people engage in to acquire new knowledge, 
skills or attitudes (step 2.2). Observing demonstrations, role playing, 
completing exercises, and constructing projects are examples of programme 
activities. Resources include people, materials, equipment, space, time, 
and money. List major required resources under step 2.3. Planning sheet 
2 will help you identify factors that may produce programme effects. For 
example, poorly maintained equipment or poor time management could be 
hindering student achievement of prograunme objectives. These and other 
factors could be measured during an evaluation. 



Measures and designs 

A process for selecting proper measurements and designs appears in 
planning sheet 3» This sheet enables you to focus on programme outcomes, 
activities and resources that are important to measure. The evaluation 
audience should be consulted when deciding what to measure. Plan to 
measure only those things that answer the questions of your audience — 
’’relative effect of modular instruction on student achievement” (step 
3.1). 



After identifying what will be measured, specify instruments that 
will collect required data (step 3-2). Knowing measurement instruments 
will help you plan data analyses. You can use formal procedures like the 
"t-test” shown in the sample worksheet or simply use sums, differences, or 
ranges. You may also choose to describe programme characteristics, 
discrepancies, deficiencies or levels of implementation. Describe your 
data analysis plans under step 3«3 

Step 3*4 requires that you specify an evalaution design. The design 
will indicate who will be measured, how and when. A design may be formal, 
informal or both. 

When only one programme can be measured, you can interpret results 
by comparing pretest and posttest results or through informal comparisons 
based on student profiles, curriculum documents, ’ inventory lists, 
personnel records, established standards or budget sheets. Examples of 
informal comparisons appear in Exhibit 2.3. 
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Steps 


PLANNING SHEET 3 
MEASURES AND DESIGNS 




3.1 


What will be measured? 






Relative effects of modular instruction on student achievement 
will be measured. 


3.2 


What instruments will be used? 






Manufacturer's assembly test. 




3.3 


How will data be analysed? 






Outcome data will be analysed using a t-test 


of differences 




between the mean score of an "experimental" 
"control" group on a pretest and posttest. 


group and a 


3.4 


What desi^s will be used? 






A true control group, pretest, posttest will be 


used. 


3.5 


What sampling strategy will be used? 
Random assignment. 
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Ebchibit 2.3 



EXAMPLE OF 


INFORMAL COMPARISONS 


Programme Evaluation Data 


Compared to 


External standards 


. Student test scores 


to 


Published test norms or 
previous test results 


. Attendance figures 


to 


Past attendance records 


. Workshop equipment 


to 


Prescribed equipment 
standards 


. Class size 


to 


Class sizes in other 
comparable workshops 


. Student performance 


to 


Published performance 
criterion measures 



Programme evaluation results are compared with external standards or 
criteria. For example, the observed health and safety conditions in a 
welding programme are recorded and compared with established workshop 
safety standards. This allows results from the programme evaluation 
(observed conditions) to be judged against an external criterion 
(established standards). This permits you to say, "compared to 
established health and safety standards, the welding programme meets 
and/or exceeds all criteria". This conclusion is more defensible than one 
that reports results without benefit of comparison to an external 
standard. 

Data are analysed with the greatest degree of confidence when the 
programme under evaluation has been compared with an alternative or 
control (C) group. Exhibit 2.4 shows four kinds of designs using 
controlled comparisons. Consider using these when planning to evaluate 
programme effectiveness. 
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Ebchibit 2.4 



EXAMPLES OF CONTROLLED COMPARISONS 



True control group, using pretest and posttest 
True control group, using posttest only 
Non-equivalent control group, using pretest and posttest 
Time series with non-equivalent control group 



Control or experimental group designs provide highly defensible 
conclusions on programme effectiveness. Credibility is increased because 
strict controls are exerted over programme variables — students, time, 
methods, teachers, resources, tests. For example, when a true control 
group design is desired, you must be able to randomly assign students to 
programmes. Other control group designs require that non-randomised 
comparison groups be available. Whenever true control and non-equivalent 
control group designs are desired, the evaluation must be planned before a 
new programme, method or material is initiated. Describe your evaluation 
design under step 3*4* 

You may also find it necessary to specify a sampling strategy (step 
3»5)* In some situations it may not be possible to measure everyone — 
there are too many people, they are in remote locations, or there is 
insufficient time or money. Therefore, a random sample is drawn from the 
population. A sample should be made as large as is possible within the 
constraints of time and money. The entire population is best represented 
by a large sample. The smaller the sample, the less likely its outcomes 
will reflect measurements taken from the entire population. Whatever its 
size, the sample must accurately represent the population (Cochran, 1973 - 
Sudman, 1976). 

Sample sizes necessary for credible representation of selected 
groups appear in Exhibit 2.5* Use this exhibit as a guide when specifying 
sample size. For example, if the number (N) of people in the population 
is 100, then the sample (S) size taken from Exhibit 2.5 is 80. 
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DETERMINING SAMPLE SIZE FROM A GIVEN POPULATION^ 



N 


s 


N 


s 


N 


s 


)0 


10 


220 


140 


0 

0 


291 


15 


1 -5 


230 


144 


' 1300 


29? 


20 


19 


240 


14$ 


1400 


30 2 


25 


24 


250 


152 


0 

0 


306 


30 


:s 


260 


155 


1600 


310 


35 


3 : 


270 


1 59 


1700 


31 3 


40 


36 


2S0 


163 


iSOO 


317 


45 


40 


290 


165 


1900 


320 


50 


44 


300 


169 


2000 


322 


55 


48 


320 


175 


2200 


327 


60 




340 


ISl 


2400 


331 


65 


5 6 


360 


186 


2600 


335 


70 


59 


380 


19 1 


2S0O 


333 


75 


63 


400 


196 


3000 


34 1 


SO 


66 


420 


201 


3500 


346 


S5 


70 


440 


205 


4000 


351 


90 


73 


460 


210 


4500 


354 


95 


76 


480 


2 14 


5000 


357 


lOu 


SO 


500 


2 17 


6000 


361 


MO 


S6 


550 


228 


7000 


364 


120 


92 


600 


2 34 


SOOO 


367 


130 


97 


650 


242 


9000 


36S 


140 


103 


700 


248 


lOOOO 


370 


150 


106 


750 


254 


15000 


375 


160 


1 1 3 


SOO 


2 60 


:oooo 


377 


170 


1 IS 


850 


265 


.'0000 


379 


ISO 


1 2 3 


900 


2 69 


4 0000 


3S0 


190 


127 


950 


274 


50000 


331 


200 


133 


1000 


278 


7 5000 


3S2 


210 


1 36 


1100 


2S5 


lOC'OUO 


334 



Note: ’’N" is population size, ’’s'' is sample size. 



Krijcio. v. ;ind Morgan. D.W. He term in ir.?'. sample size tor research 

activities. Educational and Psychological Measu^^^^ient . 1970- 30. 




BEST COP 
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Data concerns 



Collecting, analysing and reporting data are dealt with in planning 
sheet 4. Data collection deadlines are specified in step 4.1. Start by 
specifying the deadline for the evaluation report. Then estimate how much 
time will be needed to prepare the final document. The more extensive and 
complicated an evaluation, the more time will be required for report 
preparation. Two to three weeks are considered adequate to write a normal 
report . 

Subtract the number of weeks required for report preparation from 
the reporting deadline. This is the deadline for analysing data. The 
complexity of the analysis process will determine how much time is needed 
to complete this task. The scoring deadline is dependent upon how quickly 
test instruments can be scored. The time needed may vary from a few days 
to many weeks . 

Set pretest deadlines close to the end of the evaluation period. 
This will give a new programme or method maximum time to cause a programme 
effect. In most cases, pretests should be given at the start of the 
programme . 

Planning data presentation requires choosing a method that will be 
understood by the evaluation audience (step 4*2). Plan to visualise data 
with charts and graphs. This will enable you to identify trends in 
results. Simple visuals are more effective than complex statistical 
presentations. The evaluation design and measurements will determine 
whether or not you can use statistical procedures. 

Reports of evaluations must be carefully planned (step 4.3). 
Specify the manner in which results will be reported — written, verbal or 
both. Then outline the major report topics — background, data collection 
and analysis procedures, findings. An example of a completed evaluation 
report appears in Section VI. 




43 



- 3A - 



PLANNING SHEET 4 



DATA CONCERNS 



Steps 



4*1 What are the deadlines? 

1 April - pretest 

15 April - posttest 

17 April - scoring 

20 April - analysing 

11 May - final report 

4-2 How will data be presented? 



Graph data to show difference between mean scores of E 
and C groups. 



4.3 How will data be reported? 



Written document describing; 

- Programme background 

- Data collection and analysis procedures 

- Findings 

- Conclusions 

- Recommendations 
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Budget. Finally, an evaluation plan must define costs associated with 
collecting, analysing and reporting information. A budget must be 
developed ensuring sufficient funds to carry out the evaluation as 
proposed. Planning sheet 5 provides a budget outline. Use it as a guide. 
Notice that two cost columns are provided for each budget category; 
ESTIMATED and ACTUAL . First, enter estimated costs for each of the five 
budget categories — personnel (5*1), travel (5.2), test items (5*3)> 
communication (5*4), and other (5*5). As you incur costs, enter them in 
the ACTUAL column. This provides a good record of actual versus estimated 
costs. 



The evaluation tasks proposed will determine costs. The more 
complicated and time consuming a task, the more it will cost. For 
example, evaluation requiring comprehensive data collection and computer 
analyses will be costly. Evaluations using simple designs and data 
collection will cost less. Personnel costs are estimated by calculating 
time on task. The procedure follows: 

. Identify who will perform each task. 

. Estimate time required to complete each task. 

. Calculate staff cost per unit time. 



EXAMPLE 



Evaluation task 


Time on task 


Staff cost 


per hour 


Total personnel 
cost 


1 


60 hrs X 


20 


us $ 


= 1200 us $ 


2 


30 hrs X 


10 


US $ 


= 300 US $ 


3 


10 hrs X 


5 


US $ 


= 50 US $ 



Repeat for each evaluation task. 
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PLANNING SHEET 5 






BUDGET 




Steps 




Costs 






Estimated Actual 


5.1 


What are the personnel costs? 






Evaluator 


1200 




Consultant 


300 




Clerical 

Typist 

Other 


50 


5.2 


Tr a V e 1/ Lodging/ Fo od 






Transportation 

Housing 

Meals 


25 


5.3 


Test Items/Analysis 






Test purchase 


35 




Test scoring 
Data processing 


50 


5.4 


Communication 






Telephone 


40 




Postage 

Telex 


20 


5.5 


Other costs 




5.6 


Total costs $ 


1720 
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Cost estimates for travel/food/lodging, test items, and 
communications can be developed on the basis of; 

. current rates being paid or charged, 

. experience with similar, past evaluations and/or 
. information provided by others who have budgeted similar 
evaluations. 

Show your completed budget worksheet to others. In this way, you 
can check your cost estimates against those made by others, You may 
discover errors and gain insights into fiscal planning. 

People who have experienced similar evaluations may suggest ways to 
lower costs while still maintaining credibility. Experienced evaluators 
have found ways to keep costs manageable by; 

. using clerical help wherever possible, 

. purchasing rather than developing test instruments, 

employing consultants for only the most complicated tasks, 

. using already proven evaluation designs and procedures, 

. sampling, . 

. using instruments that can be machine scored, and/or 
. using data routinely collected by the programme. 

Keeping costs manageable should not overshadow the need for credible 
information. Changes made for the sake of economy could very well reduce 
the believability of an evaluation. 

You have now completed five planning worksheets. Review each 
worksheet and summarise data on the one-page evaluation plan. Attach the 
worksheets to the plan as supporting documents. Be sure to indicate, at 
the top of the plan sheet, who the responsible person(s) will be. 



Sumaary 



Good evaluations _are carefully planned and executed. A plan for 
collecting and analysing data increases the likelihood that an evaluation 
will be successful. Five aspects of an evalaution are described in a 
plan; 
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. Scope 

. Programme 

. Measures and designs 

. Data concerns 

• Budget 

Information for each aspect is collected using five planning 
worksheets. Each worksheet has a series of steps to follow. Completion 
of each step enables you to gather information necessary to complete the 
evaluation plan. A completed sample evaluation plan and five planning 
worksheets appear in this section. Copies of these worksheets appear in 
blank form in Section VI. Use them to guide your planning efforts. 




SECTION III 

INFORMATION REQUIREMENTS 
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INFORMATION REQUIREMENTS 



Information from different sources is used to evaluate various 
components of the curriculum. Some of the more appropriate information 
sources for each component are shovm in Exhibit 



Exhibit 3*1 



INFORMATION SOURCES FOR CURRICULUM EVALUATION 



CURRICULUM 


INFORMATION SOURCES 


Curriculum Planning 


. Employer opinions 
. Manpower supply and demand reports 
. Expert opinions 
. Advisory committee opinions 


Curriculum Development 


. Expert opinion 
. Curriculum standards 
. Programme documents 
. Teacher reactions 


Curriculum Materials Development 


. Materials specialist opinion 
. Commercially produced materials 
. Materials check lists 
. Teacher opinions 


Curriculum Effects on Students 


. Student reactions 
. Teacher reactions 
. Test results 

. Programme and resource standards 
. Programme records 


Curriculum Effects on Former 
Students 


. Employer reactions 
. Former student opinions 
. Employment records 
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To illustrate, consider that you are trying to determine whether or 
not students are learning curriculum content. You can get appropriate 
information about content mastery from test results, teacher grade books, 
and programme records. Student opinions and teacher reactions are also 
good sources of information. Focus your data collection on sources which 
are most likely to give you the information your evaluation audience wants 
to see in the final report. Also look for data sources that provide 
complementary information. This allows results from the programme under 
evaluation to be compared with another programme or standard. 

Before commencing the "exhibits" and the data collection work 
sheets, an "information sheet" will indicate the source and objective or 
target aim. This could be as follows: 



DATA collection GENERAL INFORMATION WORKSHEET 

Please check : 

J [ Vocational Training Centres 

J [ Adult Education/Training Centres 

J I Secondary Technical Institutes 



Please fill in: 

X Institute District: 

X Institute or Centre: 

X Principal/Director: 

X Title of Course: ... 

X Occupation covered: 
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Please check : 

Type of course 

J [ Practical work 

j [ Related technical 

J [ Other related 

Please fill in : 

X Instructor: 

X Evaluation carried out by: 

X Position/title : 

X Date: 



Tests, records and documents 



Test results, school records and published standards are good 
sources of information. Exhibit 3-2 shows examples for each of the three 
major information sources. Use this exhibit to guide your information 
collection activities. 
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Exhibit 3 • 2 



INFORMATION SOURCE DOCUMENTS 



Tests 



Teacher developed achievement tests 
Standardised achievement tests 
Interest inventories 
Scholastic aptitude tests 
Manipulative aptitude tests 
Performance tests 
Attitude assessments 

School Records 



Memoranda 
Grade reports 
Teacher grade books 
Official transcripts 

Diplomas or certificates of completion 

Logs, journals, diaries 

Textbooks, manuals, booklets 

Student worksheets 

Progress charts 

Rental agreements 

Letters 

Bills, requisitions, invoices 
. . Student assignment sheets 

Circulation files on books and materials 

Documents 



Programme promotional brochures 
Curriculum outlines 
Courses of study 

National safety and health standards 
Workshop and equipment standards 
National curriculum standards 
International workshop standards 
International health and safety standards 
International curriculum standards 
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A vast array of tests is available to help you collect information 
about curriculum effects on students. Tests range from teacher developed 
achievement tests to norm referenced scholastic aptitude tests. The 
information needs of your audience must be considered when you select 
tests as a means of providing required data. Be sure the tests you are 
considering will provide appropriate information. 

School records are another valuable source of information. They 
allow you to put together a reliable picture of what has occurred in the 
past. Exhibit 3-2 lists some of the more commonly kept records. These 
records are helpful in describing a curriculum or programme. Use them 
when it is necessary to do a descriptive analysis of a programme’s 
history. 

Standardised documents describing curriculum components are good 
sources of complementary data. They provide data that can be compared 
with results from curriculum components under evaluation. Nine such 
documents are listed in Exhibit 3*2. 



Information needs and supporting data 



Exhibit 3*3 shows the kinds of data to provide an audience when 
their information needs are known. To provide appropriate data, identify 
’’who” or "what” the audience wants to know about. Then determine 
specifically what they want to know about that person or thing. Suppose, 
for example, your audience wants to know about teachers. You must find 
out what it is they want to know about teaching staff. Assume they want 

to know whether or not teachers are of good quality. 

Checking Exhibit 3*3, you will see that when the audience wants to 
know about teachers, you provide data on some or all of the following: 

teaching experience 
work experience 
. teacher training 

. education level 

. gender, age 

special characteristics 
teaching evaluations 
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Exhibit 3*3 



AUDIENCE INFORMATION NEEDS AND SUPPORTING DATA 



IF, THEN, 

THE AUDIENCE WANTS PROVIDE DATA 

TO KNOW ABOUT ... ON . . . 



Students 



Teachers 



The school 



Instructional processes/ procedures 



Employment record of graduates 
Grade point average 
Achievement , aptitude , or 
intelligence test scores 
Courses completed 
Attendance 

Addresses, phone numbers 

Gender 

Age ■ 

Physical characteristics 
Punctuality 

Years of teaching experience 

Work experience 

Years of teacher training 

Education level 

Gender, age 

Special characteristics 
Teaching evaluations 

Enrolment total 
Class size 

Average daily attendance 
Budget 

Physical plant 

Faculty size and composition 

Teaching/learning processes 
Goals and objectives 
Course of study 
Instructipnal materials 
Supporting resources 
Equipment and supplies 
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Facilities 



Classrooms 
Workshops 
Ancillary spaces 
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Safety and health 



Personal protection 
Machine and equipment guards 
Accident reporting system 
Accident prevention programme 



Because the audience is concerned with quality, data from student 
evaluations of teacher performance might be provided to them. You must 
decide if this data best meets the information needs of the audience. 

However, this data alone is not enough. To be credible, the results 
from the teacher group under evaluation must be compared to another 
teacher group or standard. Student evaluations of teachers in a similar 
school setting could be used to compare results. Making these comparisons 
allows you to say, "compared to student evaluations of teachers in a 
similar school and setting, this school's teachers exceeded all comparison 
group mean scores on a standard instrument". This statement is more 
credible than one that only describes student evaluations of teachers 
without benefit of a comparison group or standard. 



Data collection worksheets 



What remains is to fulfil information needs in an orderly and timely 
manner. A series of data collection worksheets (Exhibits 3«4 to 3«14) has 
been included to meet this need. Each worksheet is designed to collect 
information about a specific curriculum component — students, teachers, 
instructional processes, facilities. When used in combination, they 
fulfil data requirements of a comprehensive curriculum evaluation. Used 
separately, they supply data about one component of the curriculum — 
objectives, methods or facilities. 

Whether used separately or jointly, the worksheets should be used in 
conjunction with the objectives, measures and designs stated in the 
evaluation plan. Consult your plan before starting to collect 
information. A careful review of the plan may save ^pu from collecting 
unwanted data. 
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Setting evaluation ratings criteria 



The following ticks may be used: 



S 

H 

BE 

M 



provisions exist extensively 



provisions exist to some extent 



provisions are limited 



provisions do not exist but are needed 



provisions not desirable or not needed 











- 48 - 



Exhibit 3*4 

. WHEN your audience wants information on the .... 

SCHOOL 



. YOU may need to provide information on 



CHECK 

Philosophy of school 
Aims, goals, objectives 
Administrative organisation 
Governing bodies/advisory committees 
Programmes offered 
Number and description of students 
Number and description of administrators 
Number and description of support staff 
Employment status of former students 
Budget 
Equipment 
Facilities 

Audio-visual resources 
Community served by school 
Industrial relations 
Staff development programme 
Teacher evaluation system 
■ CHECK your evaluation plan before deciding what data to supply. 
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Exhibit 3*5 

. WHEN jour evaluation audience wants information on.... 



OBJECTIVES 



. YOU maj need to provide information on . . . . 

TvT CHECK 

J [ Institutional aims 

J \_ Curriculum goals 

J [ Programme objectives 

J [ Extent to which objectives: 

fulfil training needs, 

identify students for whom the objectives are intended, 
define the competence of the students upon completing 
the course, 

define conditions under which performance will be 
exhibited, and 

define the criteria of acceptable performance. 

J [ Number of objectives by domain and learning level: 

number of cognitive objectives at the knowledge, 
comprehension and application levels, 

number of affective objectives at the receiving, 

responding, valuing and characterising levels, and 
number of psychomotor objectives at the imitation, 
manipulation, precision, and automation levels. 

. CHECK jour evaluation plan before deciding what data to supplj 
(Mager, 1967). 
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Esdiibit 3*6 

• WHEN your evaluation audience wants information on.... 

STUDENTS 



. YOU may need to provide information on .... 



CHECK 

Age and gender 
Grade point average 
Attendance 
Punctuality 
Goals 
Attitudes 

Social and economic status 

Test scores 

achievement, 
aptitude, 
intelligence, or 
performance 

Courses completed 

Physical characteristics 

Entry level behaviour 

Employment status of former students 

Satisfaction with programmes 

. CHECK your evaluation plan before deciding what data to supply. 
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WHEN your evaluation audience wants information on. . . . 



TEACHERS 



YOU may need to provide information on .... 

CHECK 

Number of teachers 
Age and gender 
Teaching experience 
Work experience 
Education level 
Years of teacher training 
Upgrading experience 
Teaching evaluations 
Professional development plans 
Extra-curricular contributions 
Special abilities or training 

Limitations 

CHECK your evaluation plan before deciding what data to supply. 
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Exhibit 3>8 

• WHEN jour evaluation audience wants information on.... 



TEACHING METHODS 



. YOU may need to provide information on .... 

TvT CHECK 

J \_ The kinds of methods used for instruction, including: 

~ lectures ^ -rojects 

- discussions - individual modules 

- demonstrations - case studies 

- simulations - on-the-job training 

- practical exercises - computer assisted 

The value of a method as a teaching and learning tool 
Appropriateness of a method for objectives 
Appropriateness of a method for students 
Teacher success in using a method 
Availability of equipment to sup: rt a method 
Availability of facilities to support a method 
Availability of time to properly use a method 
The costs of using a method 
. CHECK your evaluation plan before deciding what data to supply. 




eg 
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Exhibit 3*9 



. WHEN your evaluation audience wants information on.... 



WORKSHOP FACILITIES 



. YOU may need to provide information on .... 

Tyf CHECK 

J [ Available space 

R^tio of width to length 
of workshop 
per student 
for materials storage 
for student storage 
for equipment storage 
for assembly area 
m^ for classroom 
m for teacher planning 
Location of various areas to one another 

J [ Workshop orientation 

Location in relation to main school building 
Location in relation to ground 

J [ Floors, doors, partitions 

Floor material of workshop 
Width of corridor doors 
Width of workshop service doors 
Type of service doors 
Type of hardware on doors 
Flexibility of partitions 

J [ Visual conditions 

Window area in workshop 
Window arrangement 
Height of windows 
Type of window glass 
Artificial light source 

Light reflection of ceiling, walls, benchtops, 
furniture, machinery, floor 
Illumination level at workstations 



o 




I I Sound conditions 



Sound absorbing value of ceilings 
Sound absorbing value of walls 

J [ Plumbing 

Drinking and washing facilities 
Work sinks 

Automatic sprinkler systems 
Location of air compressor 
Distribution of gas outlets 

J [ Heating and ventilating 

Kind of ventilating system 
Heating system 
Temperature controls 
Exhaust system 

Relative humidity of workshop 

j [ Electrical system 

Power circuit system 
Branch circuits 
Overload protection 
Spare circuits 

Power cut-off safety buttons 
Fire alarm system 

J [ Health and safety 

Fire extinguishing agents 
First aid supplies 

J [ Color conditioning 

Wall color 
Floor color 
Ceiling color 
Machinery color 

. CHECK your evaluation plan before deciding what data to supply. 
(Adapted from Modern School Shop Planning , Prakken, 1982). 
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Esdiibit 3*10 

• WHEN your evaluation audience wants information on.... 



HEALTH AND SAFETY CONDITIONS 



. YOU may need to provide information on «... 



I V| CHECK 

J [ Workshop accident prevention programme 

J [ Accident reporting system 

J [ Personal protection 

head 

eye and face 

hand 

foot 

ear and mouth 
total body protection 

J [ Materials movement 

J [ Machine guarding 

J [ Hazardous materials precautions 

J [ Emergency treatment supplies/materials 

J [ Utilities safeguards 

J [ Fire safety 

J [ Ventilation 

. CHECK yo\ir evaluation plan before deciding trfiat data to supply. 
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£schibit 3.11 

. t/H£N your evaluation audience wants information on.... 

EVALUATION METHODS USED BY TEACHES 



. YOU may need to provide information on •••• 



V 



CHECK 

How student performance is assessed 
How student attitudes are assessed 
How student knowledge is assessed 



J [ How grades are determined 

J [ How student records are kept and managed 

J [ How student performance criteria are assessed 

. CHECK your evaluation plan before deciding wfaat data to supply. 
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Exhibit 3*12 

. WHEN your evaluation audience wants information on.... 
~ FINANCIAL RESOURCES 



. YOU may need to provide information on .... 

Tvf CHECK 

J [ Type of budget used 

revenue and expense budget 
time, space, material budget 
capital expenditure budget 
cash budget 
variable budget 
zero base budget 

J [ Budgetary control devices 

statistical data 
special reports and analyses 
operational audit 
personal observation 
time-event network analyses 
programme budget 

J [ Income 

J [ Expenses 

J [ Costs per student 

J [ Cost per programme 

J [ Costs per teacher 

J [ Costs per unit of time 

J [ Cost for equipment, physical plant, materials, staff, 

transportation, utilities 

. CHECK your evaluation plan before deciding what data to supply. 



o 
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Exhibit 3.13 

. UHEN your evaluation audience wants information on.... 

LEARNING MATERIALS 

. YOU may need to provide information on .... 

CHECK 

I I The kinds and types of materials used 

J 1 Appropriateness of materials to 

objectives 
student abilities 
- student interests 

student cultural background 

I I Prerequisites for material use 

I I Extent to which materials provide meaningful student 
participation 

I I Accuracy of materials 
1 1^. Technical quality of materials 
J I Accuracy of information presented 
1 I Evidence of effectiveness of materials 

J [ Equipment requirements for material use 

j [ Facility requirements for material use 

I 1 Personnel requirements for material use 

J [ Costs of materials 

J [ Strengths of learning materials 

J [ Weaknesses of learning materials 

. CHECK your evaluation plan before deciding what data to supply. 
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Exhibit 3*14 

. WHEN your evaluation audience wants information on.... 



SPECIFIC PROGRAMMES 


. YOU may 


need to provide information on .... 


TvT 


CHECK 


n 


Programme goals and objectives 


n 


Prerequisites 


n 


Target population 


n 


Content 


n 


Teaching methods 


n 


Learning materials 


n 


Facilities 


n 


Safety and health 


n 


Teachers 


..n 


Programme effects 




on in-school students 
on former students 


n 


Budget 


. CHECK your evaluation plan before deciding what data to supply. 
(Ohio State, A-11). _ 
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Exhibit 3*15 

. WHEN your evaluation audience wants information on . . . 



ADMINISTRATIVE STAFF 



. YOU may need to provide information on 

J \_ Secretarial support staff 

Total number of staff 

- Grade and classification 

- Secretarial support input in working hours, weeks or months 

- Staff costs 

J [ General administrative support staff 

- Total number of staff 

- Grade and classification 

Administrative support input in working hours, weeks or 
months 

- Staff costs 

J [ Cost control support staff 

- Total number of staff 

- Grade and classification 

Cost control support input in working hours, weeks or 
months 

- Staff costs 

• CHECK your evaluation plan before deciding what data to supply. 



Summary 



Curriculum evaluation requires information from many different 
sources. A series of data collection worksheets was included in this 
section to help you decide what information is needed and where you can 
get it. When used in combination they fulfil data requirements of a 
comprehensive curriculum evaluation. Used separately,, they supply data 
about one component of the curriculum. Whether used separately or in 
combination, they must be used in connection with your evaluation plan. 
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ADMINISTERING AND SUPERVISING THE EVALUATION 



To properly administer and supervise an evaluation you must follow 
an orderly process. This process requires that you perform five functions 
— planning, organising, staffing, activating and controlling (Bittel, 

1980). 



The major functions and supporting documentation are shown in 
exhibit The planning function precedes all others. Your efforts to 
perform this function will be aided by the evaluation plan developed in 
Section II. Efforts to organise and staff the evaluation will be 
supported by using the expense control worksheet and evaluation schedule 
presented this section. The activating function uses support information 
that has been funneled through the first three functions as well as an 
evaluation initiation checklist. A management control worksheet is used 
to support the controlling function. As you perform each function use 
supporting documents to guide your actions. 



Planning 



Planning requires definition of purpose and preparation of an 
evaluation plan. An evaluation plan increases the likelihood that the 
evaluation will be successful. Presented in Section II, an evaluation 
plan describes the scope of the evaluation, programme background, measures 
and designs, data concerns, and budget. These components give direction 
to the evaluation effort. A good plan makes your job easier as you 
administer and supervise the evaluation. 



Standards 



A set of standards has been included to help you make comparisons 
between results obtained from the programme evaluation and an external 
criteria. Appearing in Section VI, the standards are statements of ideal 
conditions which exist in successful TVE programmes. They describe a set 
of conditions in five areas of programme operations — curriculum and 
instruction, students, staff, organisation and administration, and 
physical plant, equipment and supplies. Use these ^standards to make 
comparisons between actual and ideal conditions. 
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The standards are intended only as guides for comparative analyses. 
Standards will vary according to the situation. For example, the standard 
for the number will vary according to the TVE programme, available 
resources and local practice. Use local standards whenever possible. 
Apply standards from this guide only in the absence of local ones.. 



EXAMPLE 

STANDARDS STATEMENTS 
Curriculum and Instruction 



1.1 Programme Description 

There are written descriptions for each TVE programme that describe 
who the programme is intended for, scope and sequence, and general 
outcomes. 

1 . 2 Performance Objectives 

There are statements of expected student performance for each TVE 
programme. .Each statement contains performance indicators, 
conditions under which students must perform, and standards 
criteria. 

1 . 3 Admission Criteria 

There is a set of written criteria for admission to programmes. 

1-4 Instructional Methods and Procedures 

There are written courses of study for each TVE programme. They are 
on file in the school's main office. Courses of study describe 
objectives, methods, procedures, facilities, equipment and resource 
list . 

1-5 Learning Process 

Learning resources (textbooks, manuals, handouts, booklets, tests, 
audio“Visuals and special materials) required for instruction are 
easily accessed by students and staff. 



o 
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Students 

2.1 Orientation to Programme 

There is an on-going orientation programme to acquaint students with 
TVE programmes , 

2.2 Guidance for Programme Placement 

There is a guidance programme to test, evaluate and counsel 
students. 

2.3 Records 

A student record system is maintained. 

2.4 Student Evaluation 

There is an evaluation programme to test studv^nts' progress and to 
certify attainment of competencies and/or skills. 

2.5 Licensing and Credentialing 

Students are given an opportunity to enter and complete programmes 
leading to appropriate licensing and credentialing. 

2.6 Guidance for Job Placement 

• There is a job placement programme that develops jobs, makes them 
available to students and assists students in securing appropriate 
work. 

2.7 Follov-up 

There is a comprehensive follow-up of programme graduates that 
determines the degree of relevance and success of the institution's 
training activities and job placement services. 
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Staff 



3.1 Selection and Preparation 

There is a written staff selection plan for TVE personnel. 

3.2 Salary and Promotion 

There is a published salary schedule and fringe benefits programme 
which includes a plan for evaluation and promotion. 

3.3 Professional Growth 

There is a written professional growth plan which provides for 
upgrading of technical and professional competencies. It provides 
for in-service activities, on-the-job experiences, participation in 
related professional organisations and university training. 

3.4 Business and Industry Involvement 

There is a written plan to involve teachers, placement officers and 
counselors with business and industry. 

3.5 Staff Evaluation 



There is an evaluation plan which determines the adequacy of 
professional preparation, performance and growth of each staff 
member . 
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Organisation and Administration 



4. 1 Philosophy, Purpose and Means of Providing Vocational Programmes 

There is a current written statement describing the institution's 
TVE philosophy, programmes and ancillary services provided for 
students . 

4.2 Advisory Boards 

There is a functioning advisory board for the TVE curriculum which 
advises staff on philosophy, policies and procedures. 

4.3 Committees 

The institution uses TVE programme trade and craft committees to 
enhance programmes for purposes of evaluation, community relations 
and curriculum development and revision. 

4.4 Policy and Procedures 

There is a set of written policies and procedures for the 
administration and operation of TVE curricula. 

4.5 Administrative Staff 

The institution has properly qualified and/or certified TVE 
administrators, supervisors and support personnel. 

4.6 Teaching Load 

There is a plan for determining appropriate teaching load consistent 
with the characteristics and demands of the programme being taught, 
the characteristics of the students, the nautre of the facilities 
and the needs of the teachers for non-instructional time. 

4.7 Financial Policies and Procedures 

There are written financial policies and procedures which provide 
for stable programme budgeting to supply resourcejSf. necessary to meet 
TVE objectives. 
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4.8 Community Relations and Support 

There are written community relations plans for the TVE 
programme ( s ) . 

4*9 Planning, Research and Development 

There is a written plan for continuous planning, research and 
development activities dealing with TVE programme operations, 
policies, procedures, curriculum, facilities, staff, equipment and 
budget . 

4.10 Evaluation 



There is a written plan for continuous collection of evaluation data 
about TVE programmes' operations, policies, procedures, curriculum, 
facilities, students, staff, equipment and budget'. 
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Physical Plant, Equipment and Supplies 



5. 1 Operation Plan 

There is a documented plan for the operation and use of TVE 

facilities, equipment and supplies including use manuals and 

emergency procedures. 

5.2 Maintenance Plan 

There is a plan for preventive maintenance and house-keeping 

activities for all TVE facilities, equipment and supplies, 

5. 3 Short and Long-Range Planning 

There is a plan for short- and long-range development of new 

facilities, acquisition of new equipment and supplies and 

modification of existing facilities and equipment. 

5*4 Safety and Health Conditions 

Each TVE programme's safety and health conditions meet local, 

regional and national standards. 

5« 5 Workshop Conditions 



Each TVE workshop meets local, regional and national standards 



5.5.1 

5.5.2 

5.5.3 

5.5.4 

5.5.5 

5.5.6 

5.5.7 

5.5.8 

5.5.9 

5.5.10 

5.5.11 

5.5.12 

5.5.13 

5.5.14 

5.5.15 



Number teaching periods per week =40-45 periods 
Number of students per class = 24 
Workshop use factor not to exceed 80 % 

Classroom use factor = 85^ - 90^ 

Standard workshop unit area = 200 m^ 

Workshop area per student (depending on subject & level) = 

4 m 2 to 12 m 2 

Classroom area per student = 1.6 m^ 

Workshop proportion =1:2 

Average workshop storage area (l/5 size of workshop) = 55 

Average classroom size = 7 x 7 m 

Minimum workshop ceiling height = 4 m 

Minimum classroom ceiling height = 3 ni 

Classroom lighting level = 3OO lux 

Minimum workshop task lighting level = 3OO lux 

Optimum average workshop temperature (cold climate) = 18 C® 
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Elichibit 4*1 

MAJOR FUNCTIONS AND SUPPORTING DOCUMENTATION FOR ADMINISTERING AND 
SUPERVISING EVALUATIONS 



FUNCTIONS 



SUPPORTING DOCUMENTS 
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Organising 

To be effective, evaluations must have proper resources. People, 
space, materials, equipment, time, and money are major resources. These 
resources must be organised to meet evaluation objectives (Exhibit 4.2). 



Exhibit 4*2 




ORGANISING RESOURCES 


Evaluation Objective 




Determine effectiveness of modular 
achievement . 


instruction on student 


Resources Requiring Organisation 




. Students, teachers, assistants, 

. Workshops 

. Modules 

. Audiovisuals 

. Test instruments 

Money 


clerks 



In this example, success of the evaluation depends on bringing 
people, facilities, instructional materials, teaching aids, tests, and 
money together at the proper time. These resources interact with each 
other and have an impact on the effectiveness and efficiency of the 
evaluation. 

Human resources must first be organised. They include all people, 
either directly or indirectly, involved in the evaluation process. 
Exhibit 4.3 shows examples of such people. 
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EXAMPLES OF HUMAN RESOURCES REQUIRING ORGANISATION 

People directly involved in evaluation 

. Students who are being evaluated 

. Teachers 

. Teaching assistants 

. Collaborators 

. Consultants 

. Evaluation specialists 

People supporting the evaluation process 

. Clerks 

. Typists 

. Data analysts 

. Test scorers 

. Administrators 

. Advisory committee members 



To achieve evaluation objectives, it is important to organise people 
for' the task ahead. you must convince them of the importance of 
evaluation. One of the best ways to get their support is to show them how 
they will benefit from actively participating in the tasks” ahead. 

The spaces (facilities) in which evaluations occur are another 
important resource requiring organisation. Some examples are shown in 
Exhibit 4*4* 
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Exhibit 4-4 

EXAMPLES OF SPACES REQUIRING ORGANISATION 

. Classrooms and workshops 

. for control groups 
. for experimental groups 

. Work-sites 

. Offices 

. Media and materials rooms 

. Libraries 

. Computing facilities 



The availability of appropriate spaces will increase the likelihood 
that an evaluation will be successful. 

Evaluation materials and supplies must also be organised. They 
include those items used by evaluators and students alike. Some of the 
more common evaluation materials are shown in Exhibit 4.5* 



Exhibit 4-5 

EXAMPLES OF EVALUATION MATERIALS REQUIRING ORGANISATION 

. Testing materials 

Test instruments 
. Answer sheets 

. Scoring sheets 

. Evaluation manuals/handbooks 

. New programmes, materials, methods 

. Modules 

. Curriculum materials 

. Audovisuals 

. Manipulative aids 
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Equipment to support the evaluation must be carefully organised. 
Without proper machines or instruments to score tests, evaluation 
objectives cannot be achieved. Consideration should also be given to 
special tools or machines required for implementing a new programme or 
teaching technique — computers, television, or workshop equipment. 

Time is a finite resource. It cannot be produced like automobiles, 
stored like grain, compressed like gas, speeded-up or slowed down (Cane, 
1972). Time can only be used. Therefore, it is important to use time 
very effectively to get done what is needed and meet evaluation 
objectives. Elxhibit 4-6 shows aspects of the evaluation situation where 
time management is critical. 
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Exhibit 4*6 

EXAMPLES OF KEY EVALUATION ACTIVITIES REQUIRING TIME MANAGEMENT 

Pre-evaluation activities 

Prepare evaluation plan 

. scope of evaluation 

. programme description 

. measures and designs 

. data concerns 

. budget 

. Get approvals 

. Set dates/schedule evaluation 

. Secure personnel 

. Order evaluation materials 

. Organise resources 

. Confirm all arrangements 

. Final check for people, places, things 

Evaluation activities 
. Start evaluation 

. Administer instruments as per plan 

. Collect data 

. Monitor progress/budget 

. Conclude evaluation 



Post-evaluation activities 

. Score instrxaments 

. Analyse data 

. Prepare report 

. Deliver report to audience 

. Follow-up 
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It is important to recognise that time is required for the 
preparation of evaluation materials — instruction sheets, manuals, test 
instriiments, reports. Preparation time for one instruction sheet or test 
instrument may vary from several minutes to many hours. It will vary with 
the evaluator's experience and objectives to be achieved. Consider the 
following as you organise time: 

. evaluator background, qualification, and interests; 

. evaluator experience with the evaluation design to be used 
(how recently and/or frequently has he or she used it); 

. quality and availability of evaluation materials; 

. time required to prepare tests; 

. time required to score Instruments; 

. design to be used. 

Finally, money must be organised. A budget must be developed 

ensuring sufficient funds to support the evaluation as it is proposed. A 
process for planning an evaluation budget was described in Section II. 
Use the information from that process to guide your management of 
financial resources. A sample expense-control worksheet has been included 
to help you manage the budget (Exhibit 4«7). It first appears in this 
section in completed form and again in Section VI as a blank worksheet. 
This worksheet will help you keep track of how expenses compare with 
budgeted amounts. 

A worksheet should be prepared for each major evaluation period — 
week or month. The length of an evaluation will determine whether you 
prepare weekly or monthly worksheets. Start by entering the total 

budgeted amount for each category of evaluation expense — $ 1,S50 for 
personnel. You get this figure from the budget worksheet in your 
evaluation plan. Now enter the remaining total budgeted amounts on the 
expense-control worksheet. Indicate the current period for which the 
worksheet will report expenditures — Exhibit 4. 7 reports March 
expenditures. The exhibit assumes two months have passed since the 
evaluation started. 

As expenditures are made, record the amount under the column headed 
"Current period expenses" and opposite the proper expense category — 
$ 500 for personnel. Add this figure to the previous period's personnel 
expenses to date and enter the sum under "Evaluation expenses to date" — 
$ 500 plus $ 150 (previous period expenses to date) equals $ 650. 
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Repeat this process for each of the budget categories. You can get 
a balance for each category by subtracting "Expenses to date" from the 
"Total budgeted amount" — budgeted amount for personnel $ 1550 minus 
personnel expenses to date $ 650 equals $ 900 "Balance". This worksheet 
will help you keep track of expenditures on a continuous basis. When kept 
up-to-date it will help you spot areas where there are likely to be cost 
overruns . 
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SAMPLE EXPENSE-CONTROL VORKSHEET 



Categories 

of 

expenses 


Current 

period 

expenses 

(March) 


Evaluation 
expense to 
date 


Total 

budgeted Balance 
amount 


Personnel 


$ 500 


$ 650 


$ 1550 


$ 900 


Travel/lodging/food 


5 


15 


25 


10 


Test items/analysis 


25 


50 


85 


35 


Communication 


30 


55 


60 


5 


Other costs 


— 


— 


— 


— 


Total costs 


$ 560 


$ 770 


$ 1720 


$ 950 



o 

ERIC 
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Staffing 
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This is the function where you determine exactly how many people 
will be needed to carry out each task. You will screen or interview 
people and select those who appear to be most suitable to fill the jobs. 
It may be necessary to develop job descriptions for the most critical 
evaluation jobs. Selection of competent people to fill these jobs is 
important for the credibility of the evaluation. Credibility is affected 
by audience perception of the evaluator's competence. The greater the 
perception of the evaluator's competence, the more likely evaluation 
findings will be judged accurate. 

What remains is to organise all resources in a way that promotes 
achievement of evaluation objectives. A scheduling worksheet has been 
included to help you organise resources (Exhibit 4.8). It first appears 
in completed form and again in Section VI as a blank worksheet. Use it as 
a guide for organising and staffing your evaluation. 




After Evaluation Evaluation Before Evaluation 



Exhibit 4*8 
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EVALUATION SCHEDULE 



Programme to be evaluated 



Weeks 


Start 

Date 


Due 

Date 


Tasks 


Person 

Responsible 


8 


29.06 


01.07 


Get evaluation plan approved 


B. Sato 




02.07 


15.07 


Select evaluation staff 


B. Sato 


7 


15.07 


22.07 


Order evaluation instruments 


A. Hendi 


4 


27.07 


15.08 


Check arrangements for staff, students and 


W. Dani 




r ^ 




tests 

f \ 


r 



EVALUATION 



24.08 



25.09 



Administer pretest 
Collect data 
Administer posttest 
Collect data 



A. Hendi 
W. Dani 
A. Hendi 
W. Dani 



28.08 


09.10 


Score test data 


B. Robert 


09.10 


16.10 


Analyse data 


Hendi, Dani 


16.10 


23.10 


Write evalaution report 


Hendi, Dani 


26.10 


28.10 


Deliver report 


B. Sato 


29.10 


02.11 


Conduct evaluation, debrief session 


B. Sato 
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Use your evaluation plan to get information for this worksheet. A 
good plan will specify objectives, tasks and deadlines. Follow these 
procedures as you complete the scheduling worksheet . ; 

. list all the tasks, responsible persons and deadlines required 
before, during and after evaluation; 

. list deadlines by length of time remaining prior to the start 
of evaluation; 

. specify the length of time needed to complete each task; 

. working backwards from the due date for each task, specify the 

start date; 

. determine which tasks depend on completion of other tasks; 
revise due dates and start dates as needed; 

. rearrange beginning dates so the schedule will show the proper 
order in which tasks must occur (Sauer, 1981 )• 

Upon completing the worksheet, show it to other people. In this 
way, you can check your work for accuracy. You may discover important 
items have not been adequately addressed or completely overlooked. Be 
safe, always check your scheduling activities with others. 



Activating 



Before starting the evaluation make a final check to see that 
everything and everyone is ready. A checklist for activating an 
evaluation appears in Exhibit 4.9. Use it to verify the status of your 
preparations. Items considered essential to the successful initiation of 
an evaluation are listed in the first column — staff assigned and ready, 
tests are available. 

The second column requires you to judge the status of each item. 
You must decide whether or not each item in column one is in the proper 
state of readiness to allow the evaluation to start as planned. Enter 
"READY” in column two if the item meets your expectations. if an item 
does not meet expectations, enter "NOT READY” in the second column. 
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Ebchibit 4-9 



CHECKLIST FOR INITIATING AN EVALUATION 



STATUS ACTION 

ITEMS READY, GO, STOP. 

NOT READY REMEDIATION 



Have clearances been secured? 

Has schedule been approved and 
distributed? 

Are evaluation staff assigned 
and prepared to start? 

Are consultants prepared to start? 

Are support staff assigned and 
prepared to start? 

Are back-up staff on alert? 

Are students, participants, or 
programmes to be evaluated 
prepared to start? 

Have test, materials, documents 
been secured? 

Has equipment been secured? 

Have facilities been scheduled? 

Are you prepared to start? 




At 
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Use the third column to indicate appropriate actions. If an item's 
status is judged "READY", enter "GO" in the last column and proceed to 
check the status of the next item in column one. When you judge an item's 
status as "NOT READY", enter "STOP" in column three and explain what must 
be done to change that item's status to "GO". For example, when an 
evaluation consultant cannot fulfil his assignment because of illness, you 
must "STOP" the process until a suitable substitute is found. When you 
locate the alternate, change the status of that item to "GO" and proceed 
with your countdown. Repeat this procedure until you have judged the 
status of all items on the checklist. 

At this point you direct all resources into action by issuing orders 
and instructions. You must communicate with those who will carry out 
evaluation tasks. The best kind of communications are those that combine 
the written or spoken word with action. Try to show and tell such as 
showing an aid how to score tests as well as telling her. Face-to-face 
communication is the ideal because you can see whether or not your timing, 
tone, or choice of words has been successful. Use informal talks, planned 
appointments and telephone calls to communicate with individuals. Because 
talking to one person at a time is so time-consuming also consider 
speaking to groups of co-workers. Consider forming an evaluation team or 
task force with whom you meet regularly. This approach will minimise the 
number of form letters, memoranda, and directives you will have to issue. 
All messages intended to be official, for the record, or that affect 
several people should be written. Whenever you amend previous 
directives, also document these actions in written form. Oral changes are 
forgotten or misinterpreted over time. 



Controlling 



When evaluation activities are in motion, you must keep track of how 
well they are working. You must measure results, compare them with 
expectations, make judgements of how important deviations may be, and take 
actions to bring results into line with intended outcomes. 

In theory you perform the five functions in the order listed. in 
practice, you will find cause to alter the sequence and even repeat some 
fuctions. What you do depends upon your observations of the evaluation's 
progress over time. What remains is to perform these functions in a way 
that promotes achievement of evaluation objectives. A process and 
procedures for achieving this eno follows. 
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A management control worksheet (Exhibit 4*10) has been included to 
hep you keep track of how well the evaluation is going. It appears in 
this section in completed form and again in Section VI as a blank 
worksheet. Use it as a guide in your efforts to control evaluation 
activities. 
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Ebchlbit 4*10 



MANAGEMENT CONTROL WORKSHEET 



Results 


Expectations 


Judgements 


Actions 


Plan approved 


Evaluation plan 
approved by 1 July 


On-time 


— 


Staff selected 


Evaluators and aid 
hired by 15 July 


On-time 


“ 


Tests ordered on 
25 July 


Tests ordered on 
22 July 


Late order, 
not critical 


— 


Staff & students 
ready, tests not 
delivered 


Staff, students, 
test ready by 
15 August 


Cannot give 
pretest as 
planned 


Get delivery date, 
adjust schedule 


Test received 


Test received 
28. August 


Okay to 
begin testing 


Notify staff 


Costs of tests 


Test costs $ 50 


Cost overrun 
on tests 


Charge excess to 
travel savings 


Tests admini- 
stered one week 
late 


Start testing as 
scheduled 


On-time, new 
schedule 


— 


Data collected 
as planned 


Collect data before 
& after treatment 


On-time, new 
schedule 


— 


Tests scored as 
indicated 


Data scored as 
planned 


On-time, new 
schedule 


— 


Analysis two 
weeks behind 
schedule 


Analyse data on 
schedule 


Late report, 
critical 


Move report date 
back two weeks, 
adjust schedule. 


Report two weeks 
late 


Report on schedule 


Okay to deliver 
report late 


Deliver report to 
audience 
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Follow these procedures as you complete the worksheet: 

1. List all the required evaluation tasks under the column 
labelled ''EXPECTATIONS” - evaluation plan approved, tests 
ordered. Get evaluation tasks from the scheduling worksheet 
you prepared earlier. The evaluation plan will also provide 
good information. 

2. As resources are put into action, list the actual results for 
each expectation — the evaluation plan was approved by 
expected deadline. 

3. Then compare the results with expectations. Enter your 

decisions under the column heading "JUDGEMENTS” — the 
approach was on time. 

4. If a judgement requires "ACTION", specify what kind of 

activity is required in the last column — call test supplier, 
get delivery date, move test date back one week, adjust 

schedule . 

Judicious use of this worksheet will enable you to control 
activities so evaluation objectives are more likely to be achieved. 



Summary 



.. You must perform five functions — planning, organising, staffing, 
activating, and controlling — to properly administer and supervise an 
evaluation. Planning requires definition of purpose and preparation of an 
evaluation plan. A good plan increases the likelihood that an evaluation 
will be successful. A second function requires that you organise all 
resources in a way that promotes achievement of evaluation objectives. 
You must arrange to bring people, facilities, materials, equipment, and 
money together at the proper time. Finding appropriate people to carry 
out evaluation tasks is the third function. You will need to screen and 
select qualified staff. It may be necessary to write job descriptions for 
key evaluation jobs. The credibility of an evaluation is dependent, in 
part, on the credentials of its staff. 

The activating function directs all resources into action through 
orders and instructions. You must combine the written or spoken work with 
action to communicate most effectively. Always place amendments to 
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directives in written form. Controlling evaluation activities requires a 
continuous and orderly approach. Using a management control worksheet 
will help you monitor progress. The worksheet included in this section 
will enable you to see how well things are going. 
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SECTION V 
CASE STUDIES 
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CASE STUDIES 



This section gives you a chance to prepare evaluation plans. Four 
case studies are presented. Each one addresses a different curricular 
concern: 

Case Curricular Concern 

1 Curriculum planning 

2 Curriculum materials development 

3 Curriculum effects on students 

4 Curriculum effects on former 

students 

Read each case study. Select one for which you want to prepare an 
evaluation plan. Make photocopies of the evaluation plan sheet and five 
planning worksheets found in Section VI. 

Using information from the case study you selected, complete the 
five worksheets. Then transfer worksheet information to the evaluation 
plan sheet. Check your plan and supporting worksheets against the 
examples given for the case study you chose. Your responses may be 
different from the examples. This is to be expected. Remember, good 
evaluations use comparative data. Compare responses, analyse differences 
and make decisions based on these analyses. Your responses may be as good 
as or better than the example answers. 



Case studies 





CASE STUDY 1 








School officials 


have received 


many inquiries 


about 


the 


availability of 


an agricultural 


mechanics programme 


. 


The 


school does not offer one. To 


determine whether or not 


to 


start such a programme, the director has asked you 


to 


give 


him 


information about 


this vocational 


area. This request 


was made 


on 1 September 


and the final 


written .report 


is 


due 


on 


1 December. You have been given US $ 700 to 


conduct 


the 


evaluation. 
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CASE STUDY 2 

Commercially produced individualised learning packages have 
been introduced into the metalworking programme. The packages 
address major leaning outcomes in sheetmetal layout, cutting, 
bending, forming, soldering and fastening with mechanical 
devices. Students acquire skills by making sheet metal 
projects. They require sound, slide projectors and a work 
station for^ every two students. A journeyman teaches the class 
in a 200 m workshop that has 10 fully equipped workstations. 
The curriculum director wants to know the effects of these 
packages on student achievement. This information will help 
him determine whether or not to recommend continued use of 
these packages. You have been directed to provide the director 
with a written report of effectiveness within 60 days. During 
this period there is but one class of 30 students in which to 
determing effectiveness of the packages. A sum of US $ 200 has 
been allocated for the evaluation. 



CASE STUDY 3 

Students in the woodworking progr amm e are not achieving 
objectives set forth in the course of study. They are not 
mastering the skills of measuring, cutting, forming, shaping, 
drilling, fastening and finishing. Students learn these skills 
by making wood products to sell. They are under the directio^ 
of a skilled craftsman. Instruction occurs in a 200 m 
workshop containing 15 fully equipped workstations. The 
teacher maintains that the problem is poor equipment and 
inadequate facilities. The vocational supervisor has told you 
to find out whether or not poor equipment and inadequate 
facilities are at fault. You have two weeks in which to 

provide a written report. A budget of US $ 50 has been 
approved. 
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CASE STUDY 4 



During the past four years the provincial vocational training 
board has funded an automotive repair programme. The programme 
develops skills in repair and maintenance of electrical, fuel, 
transmission, hydraulic, and mechanical systems. Students 
learn by working directly on motor vehicles under the 
supervision of a licensed mechanic. Instruction takes place in 
a 200 m workshop equipped with 10 workstations. They find 
employment as mechanics on farms, in small garages, and auto 
agencies. Because of budget cut-backs the board must make 
reductions in and even terminate funding for some TVE 
programmes . The board chairman has asked you to provide him 
with information about the automotive programme's effects on 
former students — whether or not they are employable. You 
have been given US $ 2000 and three months to complete the 
evaluation. You are expected to present your findings before 
the board and also to prepare an executive summary. 
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CASE STUDY 1 
EVALUATION PLAN 

Person(s) responsible for the evaluation: Vasamillet 

1 . Evaluation scope 

There is interest in a programme that prepares agricultural 

mechanics. While the school does not offer such a programme, the director 
would initiate one if a real need is proven. The objective of this 

evaluation is to determine the need for an agriculture mechanics 

programme . 

2. Programme description 

Appropriate documents will be reviewed to determine programme goals, 
activities and required resources. Existing resources will also be 

assessed to determine if they are compatible with programme requirements. 

3. Measures and design 

A random sample of potential students and prospective employers will 
be surveyed to determine needs. Manpower reports and curriculum documents 
will also be reviewed. Complementary information will be analysed, 

similarities and discrepancies noted and conclusions drawn. 

4« Data concerns 

Questionnaires will be sent to students and employers on 15 
September and again on IS October to non-respondents. Review of manpower 
reports, curriculum documents and existing resources will start on 15 
September. Tabulation and analysis of data will commence on 15 November. 
Bar graphs will be used to present data. The final written report will be 
delivered to the director by 1 December. 

5« Budget 

A total of US $ 700 has been set aside for evaluation. Personnel 
expenses will be US $ 270. Test items will cost US $ 30 and 
communications US $ 400. 
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Steps 


CASE STUDY 1 
PLANNING SHEET 1 
SCOPE OF THE EVALUATION 


1.1 


What programme is to be evaluated? 
Agricultural mechanics 


1.2 


Who wants to know about the programme? 
School director 


1.3 


What decisions will be based on the evaluation? 

Whether or not to start an agricultural mechanics programme 


1.4 


What information must be given to the audience? 

. Supply of and demand for agricultural mechanics 

Pool of qualified students 

. Programme goals, activities and required resources 


1.5 


What is/are -he objective(s) of the evaluation? 

To def. .mine the need for an agricultural mechanics 
programme 

To determine if the school has sufficient resources to 
implement such a programme 
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CASE STUDY 1 
PLANNING SHEET 2 
PROGRAMME DESCRIPTION 

Steps 

2.1 What are the programme outcomes? 

To be determined by assessment 

2.2 What are the programme activities? 
To be determined by assessment 

2.3 What are the required resources? 

To be determined by assessment 
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CASE STUDY 1 
PLANNING SHEET 3 
MEASURES AND DESIGNS 

Steps 

3*1 What will be measured? 

. Students and employers to be served by the programme 
. The supply of agricultural mechanics 

. The demand for agricultural mechanics 
. The pool of qualified students 

. Programme goals, activities and required resources 

3.2 What instruments will be used? 

. Student and employer questionnaires 

. Manpower supply and demand documents 

. Curriculum and facility documents 

3.3 How will data be analysed? 

. Descriptive analysis of observed agreement/differences 

between official statistics and employer and student 
responses 

. Descriptive analysis of existing resources and those 

prescribed in curriculum documents 

3*4 What designs will be used? 

. Informal comparisons of: 

manpower data and student-employer responses 
existing resources and those prescribed in 
standard documents 

3.5 What sampling strategy will be used? 




Random sample of employers and students 



CASE STUDY 1 



PLANNING SHEET 4 
DATA CONCERNS 



Steps 



4.1 What are the deadlines? 



1 September 
15 September 
15 October 
15 November 
1 December 



Develop questionnaires, order documents 
Send questionnaires, read documents 
Send second questionnaire to non-respondents 
Tabulate/analyse data 
Final report 



4*2 How will data be presented? 

Bar graphs will be used to show employer, student and manpower 
data 



4*3 How will data be reported? 

Written report delivered to the school director 








CASE STUDY 1 
PLANNING SHEET 5 
BUDGET 



Steps 

5.1 What are the personnel costs? 

Evaluator 

Consultant 

Clerical 

Typist 

5.2 Travel/lodging/food 

Transportation 

Housing 

Meals 

5.3 Test items/analysis 

Test purchase 
Test scoring 
Data processing 

5 . 4 Conmunicat ion 

Telephone 

Postage 

Telex 

5.5 Other costs 



Costs 

Estimated Actual 



$ 200 
20 
50 



30 



100 

300 



5.6 Total costs 





$ 700 



Discussion of case study 1 
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This case study addressed curriculum planning — whether or not to 
offer a TVE curriculum. To provide appropriate information a needs 
assessment was proposed. This design uses complementary information to 
draw conclusions . 




Needs Assessment 



Information is drawn from different sources — employer and student 
responses to questionnaires and published manpower data. A second 
dimension looks at proposed programme goals, activities and required 
resources in terms of existing curricula and resources. This information 
helps the director decide if the school has sufficient resources to 
implement the new programme. 

Needs assessments are highly speculative. Decisions are subjective 
because reliable evidence about the supply of and demand for workers is 
not always available. Also, changes in society, technology and the 
economy create an everchanging job market- At best, a needs assessment 
provides a snapshot in time of the employment situation (Mager, 1967). 
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CASE STUDY 2 
EVALUATION PLAN 

Person(s) responsible for the evaluation: Sano 

1 . Evaluation scope 

The metalworking programme has adopted individualised learning 
packages to teach basic sheetmetal practices. Because these materials are 
costly, there is a need to know how effective they are in helping students 
achieve objectives. Effectiveness data will help the curriculum director 
decide whether or not to recommend continued use of the learning packages. 
The evaluation objective is to determine how effective modules are in 
helping students meet objectives. 

2. Programme description 

This programme enables students to perform basic sheetmetal 
operations — layout, cutting, bending, forming, soldering and fastening. 
A journeyman teaches the course using learning packages. The packages 
require sound, slide projectors and a work station for every two students. 

3. Measures and design 

A single group, pretest-posttest design will be used to measure the 
relative effects of learning packages on student achievement of 
objectives. All students will receive an objectives-based test on basic 
sheetmetal practices. Using a programme objectives checklist, gains made 
over time will be reported by subtracting average pretest scores from 
posttest scores for each desired outcome. 

4. Data concerns 

A pretest will be given on IS duly. Seventeen days of instruction 
will be given using learning packages. A posttest will be given on 10 
August. Bar graphs will be used to present data about achievement of 
objectives. A written report will be delivered to the curriculum director 
by 31 August. 

5. Budget 

Funds to complete the evaluation equal US $ 200. Personnel will 
require US $ 120 with test items and postage requiring US $ 70 and $ 10 
respectively. 
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CASE STUDY 2 
PLANNING SHEET 1 
SCOPE OF THE EVALUATION 

Steps 

1.1 Vhat programme is to be evaluated? 

Metalworking 

1.2 Who wants to know about the programme? 

Curriculum director 

1.3 What decisions will be based on the evaluation? 

Whether or not to recommend continued use of learning packages 

1.4 What information must be given to the audience? 

Data on how effective learning packages are in helping 
students achieve objectives 

1.5 What is/are the objective(s) of the evaluation? 

To determine how effective learning modules are in helping 
students meet objectives 
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CASE STUDY 2 




PLANNING SHEET 2 




PROGRAMME DESCRIPTION 


Steps 




2.1 


What are the programme outcomes? 




Students will be able to perform the following operations; 

• layout , 

. cutting, 

. bending, 

forming, 

. soldering, and 

. fastening 


2.2 


What are the programme activities? 




Students work under the direction of a journeyman and make 
projects. They: 




. layout sheetmetal projects, 

. cut sheetmetal, 

• form metal, 

. solder metal joints, and 

. join metal with mechanical fasteners 


2.3 


What are the required resources? 




. Sound, slide projectors 

• Work station for every two students 

. Learning packages 

200 m workshop with 10 fully equipped workstations 
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Steps 


CASE STUDY 2 
PLANNING SHEET 3 
MEASURES AND DESIGNS 


3.1 


What will be measured? 

Relative effects of individualised learning packages on 
student achievement 


3.2 


What instruments will be used? 

. Objectives-based sheetmetal test 

. Programme objectives checklist 


3.3 


How will data be analysed? 

Using a programme objectives checklist, gains made over 
time will be reported by subtracting average pretest 
scores from posttest scores for each desired outcome 


3.4 


What designs will be used? 

One group , pretest, posttest using ai objectives-based 
instrument 

. Informal comparison with objectives checklist 


3.5 


What sampling strategy will be used? 
None 
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CASE STUDY 2 
PLANNING SHEET 4 
DATA CONCERNS 




Steps 


4.1 


What are the deadlines? 






1 July - Assi-: ’, a person to administer 

posttest 

2 July - Order tests and checklist 

15 July - Pretest 

10 August - Posttest 

11 August - Score tests 

12 August - Analyse/compare results 

20 August - Prepare reports 

31 August - Final report 


pretest and 


4.2 


How will data be presented? 






Bar graphs will be used to present data about the 


achievement 




of objectives. Each bar will represent one objective 


4.3 


How will data be reported? 






Written report to the curriculum director; 


to include 




findings on how effective learning packages are 
students achieve objectives 


in helping 
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Steps 


CASE STUDY 2 
PLANNING SHEET 5 
BUDGET 


Costs 


5.1 


Vhat are the personnel costs? 


Estimated Actual 




Evaluator 

Consultant 


$ 90 




Clerical 


10 




Typist 


20 


5.2 


Travel/lodging/food 




5.3 


Transportation 

Housing 

Meals 

Test iteffls/analysis 






Test purchase 


50 




Test scoring 


20 


5.4 


Data processing 
Communicat ion 






Telephone 

Postage 


10 


5.5 


Telex 

Other costs 




5.6 


Total costs 


$ 200 
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Discussion of case study 2 

Because only one group of students was available for evaluation, a 
before and after - pretest - posttest design was selected. 




Before and After Design 



Programme goals were clearly specified and easily converted into 
objectives. This allowed the evaluator to write or find tests to measure 
attainment of programme objectives. The evaluator was able to report 
student achievement by objectives, communicating to the audience what was 
achieved over the evaluation period. 

This design provides no comparative data which can be used to judge 
the quality of the learning package's results. It is not possible to 
attribute achievement of objectives to the learning packages alone. 
Achievement gains might be due to out-of-school work experiences. Failure 
to achieve objectives might be due to a shortage of tools and supplies, 
not to poor learning packages. 
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CASE STUDY 3 

Evaluation plan 

Person(s) responsible for the evaluation: Amad 



1» Evaluation scope 

Some people believe the woodworking progranune is ineffective because 
workshop equipment and facilities are deficient. The vocational director 
must decide whether or not this is true so he can take corrective action. 
It is the objective of this evaluation to describe existing workshop 
conditions in relation to documented standards and to report 
discrepancies . 

2. Prograinme description 

Students should be able to measure, cut, form, shape, drill, fasten, 
and finish wood products. Students construct wood products under the 
supervision of a skilled craftsman. Construction activities occur in a 
200 ra workshop containing 15 fully equipped workstations. 

3^ Measures and design 

A comparative analysis of existing workshop conditions will be made 
with external standards. Existing equipment and facilities will be 
measured using a self-developed checklist. Comparative data will be 
obtained from review of standards documents. The data will be compared 
and deviations from standards reported. 

4« Data concerns 

Assessment of existing workshop conditions and review of standards 
will begin on 5 April. Comparative analyses will commence on 10 April. 
Results will be presented in a three column table — with the first 
showing existing conditions, the second standards, and the third 
discrepancies. A written report will be delivered to the supervisor by 15 
April . 

5» Budget 

A total of US $ 50 has been allocated for the evaluation. Clerical 
and typing personnel will receive US $ 30. Equipment and workshop 
standards will be purchased for US $ 20. 




CASE STUDY 3 



PLANNING SHEET 1 
SCOPE OF THE EVALUATION 

Steps 

1.1 What progranme is to be evaluated? 

Woodworking 

1.2 Who wants to know about the programme? 

Vocational supervisor 
. Teacher 

1.3 What decisions will be based on the evaluation? 

Whether or not equipment and facilities are the cause of poor 
student achievement 

1.4 What information must be given to the audience? 

. Description of available equipment and workshop 

facilities 

. Description of ideal woodworking equipment and 

facilities 

. Discrepancies between existing conditions and standards 

1.5 What is/are the objective(s) of the evaluation? 

To describe existing equipment/facilities in view of ideal or 
model workshop conditions and to report discrepancies. 
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CASE STUDY 3 
PLANNING SHEET 2 
PROGRAMME DESCRIPTION 

Steps 

2.1 What are the prograue outcomes? 

Upon completion of the p^og^^^Imle, students will be able to: 

. measure, 

. cut, 

. f orm , 

. shape , 

. drill, 

. fasten, and 

. finish wood products 

2.2 What are the programme activities? 

Students produce wood products under the direction of a 
skilled craftsman 

2.3 What are the required resources? 

2 

200 m workshop containing IS fully equipped workstations 
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CASE STUDY 3 








PLANNING SHEET 3 








MEASURES AND DESIGNS 




Steps 








3.1 


What 


will be measured? 






, 


Existing equipment and workshop 






• 


Ideal or pro^ ribed equipment and workshops 




3.2 


What 


instruments will be used? 








Equipment checklist 






• 


Workshop facility checklist 




3.3 


How will data be analysed? 






, 


Descriptive analysis of existing equipment and 


facility 




• 


Descriptive analysis of prescribed or ideal 
and facilities 


equipment 




• 


Comparative analysis of existing and 
equipment and facilities 


prescribed 


3.4 


What 


designs will be used? 








Informal comparisons of existing equipment and facility 






with an external standard 




3.5 


What 


sampling strategy will be used? 






None 
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CASE STUDY 3 




PLANNING SHEET 4 




DATA CONCERNS 


Steps 




4.1 


What are the deadlines? 




1 April - Get equipment and facility standards, develop 

checklist 

5 April - Using checklist, evaluate equipment and 

facility 

10 April - Compare existing with standards 

12 April - Analyse and write report 

15 April - Report 


4.2 


How will data be presented? 




Tables will be prepared that show existing conditions in one 
coluim and prescribed standards in another. A third column 
will illustrate differences and/or similarities. 


4.3 


How will data be reported? 




Deliver written report to vocational supervisor. Document 
will describe existing conditions compared with documented 
standards . 
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CASE STUDY 3 






PLANNING SHEET 5 
BUDGET 




Steps 




Costs 

Estimated Actual 


5.1 


What are the personnel costs? 






Evaluator 

Consultant 






Clerical 


$ 10 




Typist 


20 


5.2 


Travel/lodging/food 

Transportation 

Housing 






Meals 




5.3 


Test items/analysis 

Test purchase 
Test scoring 
Data processing 




5.4 


Communication 

Telephone 

Postage 

Telex 




5.5 


Other costs 

Equipment and facility 






standards documents 


20 


5.6 


Total costs 


$ so 





Ill 



Discussion of case study 3 



In this case, the director was concerned with curriculum effects on 
students — whether or not skills were learned by students. He was told 
the curriculum was ineffective because equipment and workshop were 
deficient. To help him determine whether or not this was true, the 
evaluator gathered and analysed complementary information. 




Informal Design Using Complementary Information 



Complementary information is collected from different sources — 
existing equipment and workshop and published equipment and workshop 
standards. This information is compared and used to draw conclusions. 
The comparison may show a discrepancy between what equipment/facilities 
exist and that which should be available in a standard programme. This 
discrepancy can be used to draw conclusions and make recommendations for 
correcting equipment and facilities deficiencies. 
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CASE STUDY 4 
EVALUATION PLAN 

Person(s) responsible for the evaluation: Richy, Cline 



1 . Evaluation scope 

Faced with declining resources the vocational training board must 
reduce, and even terminate, its financial support of some TVE programmes. 
To help the board make these decisions, its chairman has requested 
information on the automotive repair programme. He wants to know whether 
or not the programme affects the employability of former students. The 
evaluation objective is to determine programme effects on employability. 

2. Programme description 

Students learn to repair and maintain electrical, fuel, 

transmission, hydraulic, and mechanical systems. They learn skills by 
repairing motor vehicles under tb^e direction of a licensed mechanic. 
Instruction occurs in a 200 m workshop with 10 fully equipped 

workstations. Former students get employment as mechanics in small 
garages, on farms, and in auto agencies. 

3« Measxires and design 

A follow-up study will collect information from former students and 
employers. Questionnaires will collect complementary information for 
purposes of comparative analyses. A second component will collect and 
report data on the number of former students employed in the field for 
which they were trained. 

4. Data concern 

Questionnaires will be mailed on 10 January and again on 1 February 
to non-respondents. The number of former students employed as mechanics 
will be presented in tabular form. An oral report will be presented to 
the board on 31 March, accompanied by a written document. 

5. Budget 

A sum of US $ 2000 has been budgeted. Personnel require US $ 1300 
while travel and communication require US $ 200 and $ 500 respectively. 
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Steps 


CASE STUDY 4 
PLANNING SHEET 1 
SCOPE OF THE EVALUATION 


1.1 


What programme is to be evaluated? 
Automotive repair 


1.2 


Who wants to know about the programme? 

Chairman and members of the provincial vocational training 
board 


1.3 


What decisions will be based on the evaluation? 

Whether or not to continue, reduce or eliminate fxinding for 
the automotive repair programme 


1.4 


What information must be given to the audience? 




Programme effects on former students; employability and 

satisfaction 


l.S 


What is/are the objective(s) of the evaluation? 

To determine the automotive programme ' s effect on former 




Students — employability 
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CASE STUDY 4 
PLANNING SHEET 2 
PROGRAMME DESCRIPTION 

Steps 

2.1 What are the prograinme outcones? 

Students will be able to repair and maintain: 

. electrical, 

. fuel, 

. transmission, 

. hydraulic, and 

. mechanical systems. 

They take jobs as mechanics in small garages, on farms and in 
auto agencies. 

2.2 What are the programme activities? 

Students learn to repair and maintain motor veichles under the 
direction of a licensed mechanic. 

2.3 What are the required resources? 

2 

A 200 m workshop with 10 fully equipped workstations. 
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CASE STUDY 4 






PLANNING SHEET 3 






MEASURES AND DESIGNS 


Steps 






3.1 


What 


will be measured? 




• 


Employement record of former students 
Employer satisfaction with former students 
Former students’, satisfaction with their job 
Former students' satisfaction with their training 


3.2 


What 


instruments will be used? 




• 


Employer questionnaire 
Former student questionnaire 
Emplo 3 nnent record 


3.3 


How will data be analysed? 




• 


Descriptive analysis of employer comments 
Descriptive analysis of former student comments 
Comparative analysis of employer and former student 
comments 

Tabulate employment rate 


3.4 


What 


designs will be used? 




• 


Informal comparisons of employer and former student 
comments 


3.5 


What 


sampling strategy will be used? 




None 
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CASE STUDY 4 
PLANNING SHEET 4 
DATA CONCERNS 



Steps 

4.1 What are the deadlines? 



1 January 
5 January 

10 January 

1 February 
20 February 
25 February 

11 March 
31 March 



Get staff, order materials, get addresses 
Develop questionnaires 
Send questionnaires 

Send second questiormaire to non-respondents 

Tabulate responses 

Analyse data 

Prepare report 

Final report 



4.2 How will data be presented? 



. Employer and former student data will first be graphed 
separately 

. Then bar graphs will be made showing comparisons of 
employer and former student responses 
. Table showing employment record 

4.3 How will data be reported? 



An oral report will be given to the board along with a written 
document. The reports will describe programme effects on 
former students. 
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CASE STUDY 4 
PLANNING SHEET 5 
BUDGET 



Steps 



Costs 

Estimated Actual 



What are the personnel costs? 

Evaluator 

Consultant 

Clerical 

Typist 

Other 



$ 500 
500 
100 
100 
100 



5-2 Travel/lodging/food 

Transportation 200 

Housing 

Meals 

5-3 Test items/analysis 

Test purchase 
Test scoring 
Data processing 

5 . 4 Communication 

Telephone 200 

Postage 300 

Telex 

5*5 Other costs 



5-6 Total costs 




$ 2,000 
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Discussion of case study 4 

The vocational training board asked about curriculum effects on 
former students — whether or not the curriculum affects the employability 
of former students. To answer this question a follow-up study was 
designed. 




(Ohio State, A- 10) Follow-up Design 



Complementary information is collected from former students and 
their employers. Their responses are compared and conclusions drawn. A 
second component collects and reports data on the number of former 
students employed in the field for which they were trained. 



Summary 

Four case studies were presented in this section. Each one 
addressed a major curriculum concern — planning, materials, and programme 
effects on in-school students and former students. You were given a 
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chance to prepare an evaluation plan for one or more cases. Examples of 
completed plans enabled you to compare your work with that of another 
person. Discussions following each example plan pointed out the strengths 
and weaknesses of each solution. There were no right or wrong approaches 
to the cases. However, some are better in one situation than in another. 
Situational constraints dictated which designs and methods were most 
appropriate. 





SECTION VI 

CHECKLISTS, WOKKSHEETS, 
AND 

VOCABULARY 
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GLOSSARY 



Activating* Directing resources into action by issuing orders and 

instructions . 



Aim. 



Assessment . 



A foreseen outcome that gives direction to an activity 
and motivates behaviour (see Goal). 

The process by which one attempts to measure the quality 
and quantity of learning and teaching using various 
techniques . 



Assessment, needs . Determining the needs and priorities of a person, group, 
or organisation. 

Behaviour. Any observable act. 



Case study. 



Categories . 
Community. 



The presentation of a true or simulated situation to 
develop the judgement of students who evolve and prepare 
possible solutions. 

Classes to which certain statements uniquely apply. 
Synonym for the general public. 



Competency. Ability to control the environment — job tasks, 

personal life, financial affairs. 



Control group. Research team used for group of individuals who match as 
near as possible, a similar group which is the subject 
of experimental research or evaluation. The effects of 
any change in an experimental group treatment is 
measured against the base line of the control group. 

Control group, non-equivalent or non- ramuomised. 

A group similar to the experimental one, but not formed 
by random assignment. 
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Control group, true or equivalent. 

A group formed by random assignment. 

Controlling. Measuring and correcting activities to assure that 

events conform to the evaluation plan. 

Cooperative work experience. 

A programme that provides for alternation of study in 
school with a job in business or industry. 

The extent to which evaluation information is 
believable . 

Recognised standard or level of performance against 
which individual scores may be compared. 

A training design or plan that defines: aims, goals 

and objectives of an educational activity; the ways, 
means and activities used to achieve desired outcomes; 
and procedures and instruments needed to evaluate 
actions. 

Curriculum development. 

Course of action design to produce a set of learning 
experiences . 

Curriculum materials . 

All printed matter , audio-visuals and manipulative aids 
used for instructional purposes. 

Developmental stages, curriculum. 

Period in a curriculum's evolution characterised by a 
specific cluster of components or by the completion of 
specific tasks. 

Students of academic ability who leave TVE programmes at 
the earliest opportunity before achieving their 
potential. 

The total process developing human ability and 
behaviour . 



Early leavers. 



Education. 



Credibility. 



Criterion. 



Curriculum. 
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Education technical. 

Educational preparation for occupations between the 
skilled trades and the professions. 

Education, vocational. 

A programme of education below college grade organised 
to prepare the learner for entrance into a particular 
vocation or to upgrade employed workers. 

Education, vocational adult. 

Instruction below the college level intended to prepare 
the student for work in an occupation that requires 
technical skill and knowledge. 

Education, vocational-technical . 

Instruction below the college level intended to prepare 
the student for work in an occupation that requires 
technical skill and knowledge. 



Effect, programme. 

The effect of an experimental factor or treatment under 
controlled conditions — when other factors are held 
constant . 

Evaluation. A decision enabling process. 

Evaluation design. 

A plan that describes the who, what, where, when and how 
of an evaluation. 

Evaluation instrument . 

A test or procedure used in evaluation. 

Evaluation plan. A framework for collecting, analysing and reporting 
data. 

Experimental group. 

A group of people who all receive the new or 
experimental material, technique or programme. 

Faculty. The teaching staff of a school, college, institute or 

university. 
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Gender . 


The sex of a person — male or female. 


Goal. 


An aim or purpose. A general statement of an outcome or 
end to be attained in any behaviour situation. (See 
outcomes) . 


Group . 


A collection of people defined by the treatment or 
programme they receive — experimental group or control 
group. 


Instruction. 


See teaching. 



Instruction, individualised. 

Differentiation of instruction according to individual 
differences in students. 

Interest inventory. 

An instrument used to obtain information on the personal 





and/or leisure interests of students. 


Mastery learning. 


Concept in which mastery of a topic is possible for all 
individxials provided that each learner is given the 
optimum quality of instruction which is appropriate for 
his/her make-up and that each learner is given- the time 
he/she needs. 


Measurement . 


Act of finding dimensions of any object and the quantity 
found by such an act. 


Model. 


Representation of a concept or system in a two or three 
dimensional diagram or a mathematical or other analogous 
form. 


Module course. 


Unit of work in a course of instruction which is 
virtually self-contained. 


Module training. 


Method of training based on the concept of building-up 
skills and knowledge in discrete units. 



Norm-referenced test. 

A test which enables an individual’s performance on that 



o 
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instrument to be compared with normal or average 
performance for a whole group of similar individuals. 
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Objectives, behavioural. 

Specific statements of observable behaviour which a 
learner displays. 

Objectives, evaluation. 

Specific statements of observable outcomes which an 
evaluation produces. 



Objective test. Test or item in a test which can be scored objectively. 



Occupation. 



Organising. 



Outcomes . 



(1) Person's trade, vocation or principle means of 
earning a living; 

(2) Group of jobs with a significant number of tasks 
and skills in common. 

A process that involves determinig activities required 
to achieve goals, grouping these activities, assigning 
people to do them, delegating authority to carry them 
out, and coordinating these actions. 

An end or goal to ' be attained in any behaviour 
situation. (See goals). 



Performance. 

Philosophy. 

Planning. 



Actions of a person or group when given a learning task. 

A point of view that serves to guide the school’s 
actions and thinking.’ 

A decision making process involving the selection of 
courses of action to be followed. 



Planning sheet. A prepared form to aid evaluators in organising their 
work effectively. 

Posttest. Test administered at the end of a programme to check the 

effectiveness of the programme and/or the performance of 
students. 

Pretest. Test given to a student before taking a training or 

education programme to test his/her suitability for it 
and , by comparison with a post test , to gauge the 

programme's effectiveness. 
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Programme . 



Random sample 



Randomisation 



Range . 



Ranking . 



Rank order. 



Ratings . 



Ratio . 



Resources. 



Sample. 



Sampling . 



o 




One component of a curriculum that addresses an 
instructional area or course — welding, child care, 
electronics. 

(sampling) . 

Individuals chosen for study in a way tha guarantees 
equal likelihood of their having been picked for all 
possible samples of similar size that might be selected 
from the members of the population involved. 

A means of assuring sameness or representativeness of 
assignment or selection. 

Statistical term to indicate the difference between the 
highest and lowest values on a set of measurements. 

To distribute scores or items in order of merit or 
magnitude . 

Listing of students or items in order of merit of 
attainment or other agreed attributes. 

Ranking students or items on a particular characteristic 
as high, medium or low. 

Indicated quotient of two numbers which show the 
measures of two similar quantities. 

People, space, materials, equipment, time and money are 
major resources. 

A group which is selected from a larger group or 
population for the purpose of examination with a view to 
making generalisations about the population as a while. 

Statistical term for the general process of selecting a 
sample from a study population. In general, sampling 
attempts to extract random samples. In much educational 
evaluation, randomisation is not possible. Two 
principal classes of non-random sample are probability 
samples which are partially random and non-probability 
samples which have no random element. 
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Scholastic aptitute test. 

Predictive tests used to evaluate prospective students. 

Scheduling. Planning and organising of evaluation time and schedules 

in the school. 

Standard. Any criteria by which people or things are judged. 

Standardised test. 

A test for which content has been selected and checked 
empirically, for which norms have been established and 
for which uniformed administration and scoring exist, 
and which may be relatively objectively scored. 

Standard works. Books which are recognised and regularly used as 
authoritative sources of information in their subjects. 

Strategies. Techniques used in learning or thinking. 



Statistical significance. 

Where an occurance can be predicted on statistical 
evidence and there is low probability of it taking place 
on the basis of chance alone. 

Target group. That group of people for which evaluation information is 
being collected. 

(to) Teach. To impart knowledge or skill to another; to give 

instruction to another; to educate or to train another; 
to facilitate learning. (See training). 

Teacher. One who teaches, especially a person employed by a 

school to teach. 

Teaching methods. Standard procedures in the presentation of instructional 
material and content of activities. 



Teaching techniques. 

Specific ways of presenting instructional material or 
conducting instructional activities. 
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Time series 



Training. 



t-test . 







design. 

An instrument related to the programme that will be 
started is administered at regular intervals before the 
programme begins, during the programme and after it 
ends . 



Systematic practice in the performance of a skill. (See 
education) . 

A way of testing the statistical difference between two 
means . 
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EXAMPLE EVALUATION REPORT 



Evaluation Report for: 

Prepared by: 

Submitted on: 

Programme or Process Evaluated; 
Submitted to: 



Tara Vocational School 
B. Wila 

06.07.84 

Curriculum Objectives 
Vocational Training Board 



Background 

To determine whether or not to propose changes in curriculum, the 
vocational training board chairman requested information on programme 
objectives. He wanted to know if each programme's objectives addressed 
industrial training needs. The objective of this evaluation was to 
determine, for each vocational programme, which objectives addressed 
training needs. 



Data collection and analysis 

A discrepancy analysis was used. Objectives of each vocational 
programme were compared with counterpart industry training needs. Results 
of the comparisons were analysed for similarities and differences. 
Discrepancies between programme objectives and industry needs were 
recorded and reported. 
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Findings 

NuBber of Discrepancies Between Industrial Training Needs 
and Objectives by Progranme Areas 



Vocational Programme 


Number of 
objectives 
meeting needs 


Number of 
training 
needs 


Discrepancy 


Agriculture mechanics 


13 


15 


2 


Business practices 


10 


20 


10 


Construction 


15 


20 


5 


Food preparation 


12 


19 


7 


Health care 


15 


21 


6 


Totals 


65 


95 


30 



Thirty instances were found where vocational curriculum objectives 
did not address industry training needs. The greatest number of 
discrepancies were found in the business practices programme (10). The 
next highest number (7) was found in the food preparation programme. 
Health care, construction and agriculture mechanics each accounted for 6, 
5, and 2 discrepancies respectively. 

Conclusions 

Vocational curriculum objectives are not meeting all of industry's 
training needs. The greatest discrepancy between industry needs and 
objectives appears in the business practices programme. Discrepancies are 
also evident in the school's four other programmes. 

Recoaaendations 

1. Initiate continuing studies of the changing nature of industry 
training needs. 

2. Revise curriculum objectives starting with business practices. 
Section I 
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EVALUATION PLAN 

Person(s) responsible for the evaluation: 

1 . Evaluation scope 



2 . Programme description 

3* Measures and design 



4. Data concerns 

S* Budget 



Section II 
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PLANNING SHEET 1 
SCOPE OF THE EVALUATION 



Steps 

1.1 What programme is to be evaluated? 

1.2 Vho wants to know about the programme? 

1.3 What decisions will be based on the evaluation? 

1.4 What information must be given to the audience? 

1.5 What is/are the objective(s) of the evaluation? 



Section II 
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PLANNING SHEET 2 
PROGRAMME DESCRIPTION 



Steps 



2.1 What are the programme outcomes? 



2.2 What are the programme activities? 



2.3 What are the required resources? 



Section II 
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PLANNING SHEET 3 
MEASURES AND DESIGNS 



Steps 

3-1 What will be measured? 



3-2 What instruments will be used? 



3*3 How will data be analysed? 



3-4 What designs will be used? 



3.5 What sampling strategy will be used? 



Section II 
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PLANNING SHEET 4 
DATA CONCERNS 



Steps 

4-1 What are the deadlines? 



4.2 Hov %rill data be presented? 



4-3 How will data be reported? 



Section II 
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Steps 


PLANNING SHEETT S 
BUDGET 


Costs 


s.l 


Vhat are the personnel costs? 


Estimated Actual 


S.2 


Evaluator 

Consultant 

Clerical 

Typist 

Travel/lodging/food 




S.3 


Transportation 

Housing 

Meals 

Test items/analysis 




S.4 


Test purchase 
Test scoring 
Data processing 

Coomunication 




5.5 


Telephone 

Postage 

Telex 

Other costs 




5.6 


Total costs 




Section II 


0 
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EXPENSE - CONTROL WORKSHEET 



Current Period Evaluation Total 



Expense Expenses Expenses to Budgeted Balance 

Categories • ( ) Date Amoxint 



Personnel 

Travel/lodging/ 

Food 

Test items/ 
analysis 

Communications 

Other 



Total costs 



Section IV, Exhibit 4.7- 
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After Evaluation During Before Evaluation 
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EVALUATION SCHEDULE 



Programme to be evaluated 




Section IV, Exhibit 4-8 



MANAGEXENT CONTROL WORKSHEET 



Results Expectations 



Judgements 



Section IV, Exhibit 4-9- 




Actions 
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